Tag: web development

Posted on

How Twitter Runs And Runs And Runs

Everybody has this idea that Twitter is easy. With a little architectural hand waving we have a scalable Twitter, just that simple. Well, it’s not that simple as Raffi Krikorian, VP of Engineering at Twitter, describes in his superb and very detailed presentation on Timelines at Scale.
If you want to know how Twitter works – then start here.It happened gradually so you may have missed it, but Twitter has grown up. It started as a struggling three-tierish Ruby on Rails website to become a beautifully service driven core that we actually go to now to see if other services are down. Quite a change.

via High Scalability – High Scalability – The Architecture Twitter Uses to Deal with 150M Active Users, 300K QPS, a 22 MB/S Firehose, and Send Tweets in Under 5 Seconds.

Read the article for a good summary of how Twitter runs. As noted, it isn’t all that easy anymore. The entire 38 minute talk is worth listening to, especially for anyone with an interest in designing next generation web apps.

One of the key points in the talk is that Twitter isn’t really a web site, it’s really an API with a web application built on top. The work is in getting the API to run as fast and as effectively as possible. The tech used to accomplish this is interesting because it isn’t just a bunch of database tables, and it is the future of the interactive web.

Posted on

When You Want to Browse the Web Without a Browser

Sometime you need to get stuff from the web without that pesky browser getting in the way. Screenshots, testing, archives, scraping, and such often call for getting a page from the web and doing something with it. Automating this with a conventional browse is no fun. Linux folks have the advantage of wget and cURL running from the command line. Programmers and developers will use the language and libraries of their choice. But it is still a chore. Well, there may be an easier way.

I’ve found a couple of javascript libraries, PhantomJS and SlimerJS (yes, very ghosty!) that provide tools for building JavaScript that can manipulate web pages, effectively browsing the web without the browser. PhantomJS “is a headless WebKit scriptable with a JavaScript API. It has fast and native support for various web standards: DOM handling, CSS selector, JSON, Canvas, and SVG.” SlimerJS “is useful to do functional tests, page automaton, network monitoring, screen capture, etc. SlimerJS is similar to PhantomJs, except that it runs Gecko, the browser engine of Mozilla Firefox, instead of Webkit.”

I’ve been looking for a way to generate screenshots of pages that I’ve generated shortened URLs for with my shortener figuring that it would be nice to have a browsable library of pages. Either of these libraries will do the trick.

You can find both on Github: SlimerJS on GH & PhantomJS on GH.

Posted on

Notes from Drupalcamp Atlanta 10/27/12

These are my notes from dcATL.
  • Josh Clark @globalmoxie
  • The mobile future
  • Mobile is a new platform. What do we do with the new platform?
  • How do we do more with mobile?
  • Sensors give us super powers.
  • Mobile provides the opportunity to interpret the environment, think of augmented reality.
    • Think of ways to use camera and audio in classroom, like prof mentions case and it pop ups on device.
  • Table Drum app usess augmented audio.
  • AnyTouch turns everyday objects into interface objects.
  • Leap Motion moves touch interface into 3d space, natural gestures.
  • Natural gestures are the next break through in interfaces.
  • We need to design for natural gestures.
  • Windows 8 is intended to work with any input interface. Hugely challenging.
  • Medical field is using all sorts of special sensors with mobile devices to drive data collection.
  • Personal sensors make sense of our environment.
  • But we don’t need more operating systems, interfaces.
  • Remote control is an answer.
  • Ambiguous control among devices is coming, think of phones in cars. Your car rings. When you park the car, the interface follows you. Migrating interface.
  • http://bit.ly/day-glass– A day made of glass from Corning.
    • One smart device somewhere that is driven by ambiguous interfaces
  • Wii U
  • Grab Magic http://bit.ly/grab-magic
  • http://bitly.com/proto-gestures
  • Sifteo cubesare social toys.
    • Download software as it needs it.
  • Web is just in case, everything is loaded in case we need it. Needs to move to just in time, software loaded when we need it.
  • Passive interfaces just work on their own, doing the things they need to do to perform the functions they are designed to do.
  • Devices will get both dumber and smarter.
  • Metadata is the new art direction – Ethan Resnick @studip101
  • A cloud of social devices
  • Look beyond the interface, beyond the device, the presentation to the content and the services.
  • Push sensors
  • Think social not FB
  • Your ecosystem
  • We’re all cloud developers
  • Mind your metadata
  • New input methods
  • The future is here
  • Eric Webb @erikwebb
  • See slideshare
  • Evaluating modules
    • Supported version, maintainer rep, usage, # of open issues, usage over time.
    • Record before and after install using Devel module
    • Search for tag ” performance ” to weed out general issues.
    • What to look at
      • When does it run?
      • How does it scale?
      • What if it fails?
      • Does my site care?
      • Do I need this module?
    • ID the problem
    • Where problems occur
      • Page building like views and panels
      • External web services
      • Overall complexity
        • Views in panels in panels….
      • Misconfigured components
    • Keep records, establish a metric, adopt a definition of done, don’t hide behind infrastructure
  • Types of caching
    • App level caching is not really configurable. Tings like menus, forms
    • Component level caching, user facing stuff like blocks, views, panels
      • Best to speed up for authenticated users
    • Page level caching is important mostly for anon users
  • Configuring Drupal

  • Randall Kent @randallkent rkent@sevaa.com
  • http://bit.ly/dcatl-services
  • Web services as the tip of the iceberg.
  • REST is the key to getting at the stuff in Drupal. REST is one way to create an API on Drupal.
  • REST
    • built on http
      • GET, POST, PUT, DELETE
    • Client/Server
      • Separates ui from data storage
    • Stateless
      • All info necessary to process request must be included in the request itself
    • Cacheable
    • Layered
    • Uniform interface
  • /myapi/node – gets XML
  • /myapi/node.json – get JSON
  • REST console for Chrome
  • http://github.com/randallkent
    • DrupalREST.php
    • DrupalREST.net
  • See http://drupanium.org
  • David Bassendine @dbassendine
  • Open data, social, business tools
  • Few modules for consuming services
  • Always start with looking on line for a module
  • REST vs SOAP
  • Get to know the API you are working with
    • URL and path structure
    • Testing in browser for GET, POST requires extension/plugin
  • Services client for D7 will consume Services from another Drupal instance
  • REST API and Query API handle some RESTful APIs that serve json
    • See red mine module for example
  • Core HTTP API for other services
    • drupal_http_request($url,$options(headers,methods,data))
    • Slightly diff D6 & D7
  • Last 2 require custom modules to do the work
  • Krumo – http://krumo.sourceforge.net/

  • Talking to Web Services – Resources

  • Matthew Connerton @connerton
  • AJAX allways for there fresh of data in the browser page with refreshing the whole page.

    Sample code for AJAX in Drupal7
  • Replaces AHAH, which is a good thing. Pulls lots in crooks stuff
  • “use-ajax” class
    • drupal_add_library(drupal.ajax) to get Ajax in.
    • Pulls jquery in
  • $form[‘#ajax’]
    • drupal_add_library(drupal.ajax) to get Ajax in.
    • Blur is the default trigger.
  • It’s may ease the pain of the auth code stuff.
  • Check Drupal API for AJAX Framework docs.
    • includes/ajax.inc
  • Using #states in Form API
  • Ctools modal to open modal boxes for editing and such.
    • “ctools-use-modal” class
  • Doug Vann dougvann.com
  • Module filter is cool
  • DraggableViews
    • Makes rows of views draggable
    • Can be rearranged by drag and drop
    • Has AJAX
    • No relationship required
    • Could use this to provide a sort on Lesson topics based on order in the topic grid
    • Use this to rearrange stuff on the topic list view itself on the home page
    • No subsets or at least not easily handled
  • Nodequeue
    • Collect nodes in an arbitrary order
    • Requires relationship in order to bring stuff into proper scope


Posted on

The HuffingtonPost Provides Open Source API For Public Polls | opensource.com

The initial release is big. It includes more than 215,000 responses to questions from more than 13,000 polls, which the HuffPost Pollster team has organized by subject and geography into more than 200 charts. Per their announcement, “the data feeds operate in real time, so shortly after we add a new poll to our database, itll appear in the HuffPost Pollster APIs responses and calculations.”Adding to the coolness is that the effort relies heavily on open source tools. The HuffPost Pollster team is publishing the data as an HTTP-based application programming interface, or API, with JSON and XML responses. They are releasing the data under a creative commons license.

via The HuffingtonPost releases Pollster, open source API for public polls | opensource.com.

This lets developers get access to a large body of polling data from over 13,000 polls. The API provides JSON and XML responses to queries sent over HTTP allowing developers to parse and display the information in their applications. This represents a major open source resource in the political field.

Posted on

Blitz – Recommended for Drupal Testing

In the spirit of continuous integration and making load testing a fun sport, our paid plans have just two simple dimensions: How much you want to scale out and how long you want to rush. We do not impose any limits on how many times you run the load tests.You can easily increase your concurrency from 250 to 750 users on the free plan simply by inviting your friends to blitz. Well add +25 referral credits to both of you when your friend signs up!

via Blitz – Making load and performance testing a fun sport.

GIve it a try at http://blitz.io/gcb7uX6lS5oXx.

Posted on

Please Make PDFs Go Away…

Most law firms have a history of using Adobe’s Portable Document Format (PDF) to distribute their brochures, papers and longer written pieces. That practice matches what web usability experts have long advised: “PDF is great for distributing documents that need to be printed,” but not much more than that. The well-traveled rule is that if a document contains more than five pages of text (hint: that excludes lawyer profiles), then PDF format is worth considering.

Now, let’s throw a wrench into this. As we approach the end of 2011, many firms and their their clients are moving toward paperless offices. Clients are consuming law firm publications on a variety of devices, including smartphones, tablets, e-readers, and large multiple-monitor desktop environments. So how likely is it that we consume a PDF on printed paper? Not very.

Slaw – Revisiting PDFs for Law Firm Websites & Mobile Publishing

Finally someone has something useful to say about the future of PDFs. As someone who has to deal with found PDFs from all over the web, I can honestly say I wouldn’t miss them if they disappeared tomorrow. PDF is an excellent way to capture the artifact of the document page, but a PDF is not a web page, and PDF is not open data. PDF is a photocopy, a snapshot picture of a document. If you are interested in doing things like indexing data, repurposing data, reusing data, then a PDF is pretty useless.

 

 

 

Posted on

Drupal in the post-page era | GarfieldTech

http://www.garfieldtech.com/blog/drupal-post-page-era

Drupal today is very page-oriented. Every request that comes in is responded to with a full HTML page. It is possible to return something else, and finally in Drupal 7 there is, sort of, native support to do so with delivery callbacks, but by and large any non-page response is an after thought at best and a hack at worst. The entire system is built around the assumption that we’re returning an HTML page. Why else would we still load the theme system and form system for an auto-complete callback?

We’re moving to a future where what we see on the web is a serious of HTML fragments tailored to our preferences or prior history. Drupal needs to move past its page orientation to function efficiently on the future web.
Posted on

Getting jQuery Right From The Start

jQuery has changed the way we write Javascript by abstracting out much of the painful cross-browser implementation details that used to plague developers, but to use it correctly still requires a little knowledge about what’s going on under the hood. In this post we’ll take a good look at jQuery’s selectors and how to use them efficiently.

Wealthfront Engineering: jQuery the Right Way.

One of the best explanations of how jQuery works and how to use it properly I’ve seen. Best part is the inclusion of things that will actually slow jQuery down, resulting in a poor experience for your visitors, coupled with faster alternatives.

Posted on

Tutorial on Building iPhone OS and Android Web Apps With AJAX

Developing for mobile devices has been a high cost, low return proposition for many years, despite the hype around it. The latest generation of smartphones powered by the iPhone OS and Google’s Android provide a much simplified solution: just build Web applications. This gives you a one build for all devices approach, which can lower the cost. Even better, these high-end devices all offer ultra-modern browsers supporting advanced HTML, JavaScript, and CSS. In this article, learn how to build Asynchronous JavaScript and XML (Ajax)-heavy applications that take full advantage of the capabilities of modern smartphones. You will learn not only how to get the most out of these devices, but also how to deal with the subtle differences between them.

via IBM developerWorks: Create Ajax applications for the mobile Web.

Good tutorial. It starts out with a trivial Java app to create content, but any other server side tech, like PHP, could be used. The mobile portion relies on HTML 5 and AJAX in iPhone OS 3.x+ and Android 2.x+ browsers. You’ll need the most recent Adroid and iPhone SDKs to put this together.

This article provides a good starting point to start looking at how to build basic mobile apps.