Pretext

This article is being written in lieu of demand from several users, developers, and other fans of the site, as well as to provide, and share, my knowledge with all who can benefit from it. Streetread was built using software that many contributed their free time to, without any compensation, and if there is anything I can do to reciprocate, besides offer this site for free, it would be this. Before we get started, I want to make the following few things clear: I am by no means an expert developer. I'm 21 years of age and have only been programming for 5-6 years. Techniques and other advice shared may not be the best solution out there. Prior to Streetread, I had absolutely no Javascript or jQuery experience whatsoever, and these were the first complete Drupal modules I've coded from scratch.

If there is one thing I can suggest to fellow developers it would be to plan as far ahead as possible before even beginning a project. Streetread was originally an extremely complex, full-featured finance site, with seven custom Drupal modules, etc, until several issues and revelations came about, forcing the site to be the way it is now. Even though I'm happier with the way it turned out, a lot of time and effort could have been saved with simple planning.

The entire site, complete planning, programming, and design, took approximately four months to complete. I would estimate development time to be between 1.5 to 2 months if six Drupal modules and three themes weren't wasted. All of the work was done by myself.

For the article reference, Streetread is currently running on Drupal 5.7 (PHP-based content management system, default installation with the exception of replacing misc/jquery.js with the following) and jQuery 1.2.6 (Javascript library).

What is Streetread?

Some readers may be unaware of what Streetread is or what it does. Streetread automatically aggregates the latest headlines from over 20 of the leading finance sites on the web, as well as all of the stocks you choose to follow, and all within a single-page, easy to use, auto-loading interface. The site aims to dramatically simplify the daily activities people undergo in order to stay updated with Wall Street activity, in terms of news and data. The goal of development was to make the site as easy as possible to use, while making the interface similar to that of a regular desktop application.

Reader Layout (Drupal, HTML)

Most people are quite stunned to see that Drupal can produce an interface such as the main page here. Drupal's extensive, easy to use modular / hook system allows you to do virtually anything. Although the theme of this site looks complex, it really is quite simple - the HTML/CSS produced for the interface is where there lies some complexity.

The main layout of the theme, residing in page.tpl.php, looks something like:

<div id="header">
 ...
</div>

<div id="content">
<div id="main">
 ...
</div>
</div>

<div id="footer" class="footer">
 ...
</div>

To toggle between full-screen and fixed-width, simple PHP IF statements are used to determine whether or not the main page is being displayed. If it is, an extra class is added to header, main, and footer, overriding the width to a fixed amount.

Now, using Drupal's hook_menu, you specify which function will handle the output for the main page; it's this simplicity that enables you to create all types of insane interfaces like Streetread. This page handler fills #main with all of the HTML needed to produce the main interface, and being that it's PHP, it is completely dynamic.

Instead of endlessly explaining all of the HTML elements used to produce the interface, I'd recommend downloading Firebug and just seeing for yourself.

Reader Layout Continued (jQuery, CSS)

Auto Dimensions:
Maintaining a perfect full-page layout is very important for the main interface. If the content area were to extend too far, veritcal scroll bars would be needed - and that would ruin the usability. Absolute positioning was the original tactic used to achieve this (by settings the content DIV to left:0, right:0, bottom:0). But, Internet Explorer 7 tends to hate simple solutions to things. For whatever reason, IE was showing extremely strange behavior with overflowed DIV's positioned in a such a manner (scrolling was near impossible, and the box tended to collapse sometimes). To avoid this headache, jQuery is used to set the height of the content section to the height of the visible window subtracted from the height of the header and slider section. Whenever the window is resized, this calcuation is performed again.

Toggle Stock Chart Section:
There are two main DIV's in the content section: #contentContainer (feed headlines) & #quoteContainer (stock chart and links). #quoteContainer is floated right, set with a fixed width, and hidden by default. #contentContainer is set with an auto width, that way once #quoteContainer is displayed, the width will constrict to fit both. Upon clicking on a stock symbol or website logo, the LI ID is used to determine whether or not to display the #quoteContainer. Displaying the DIV is as simple as:

var quote = $('#quoteContainer');
if(!quote.is(":visible")) {
  quote.show();
}

Embedded / Overlaid Stories (jQuery, CSS)

This is probably the feature of Streetread most users seem to be impressed with. If you haven't yet noticed, clicking on a link to either one of the story headlines, or stock-related links, opens the given website literally on top of the interface, along with the panel that drops down from the top. As I was using Streetread prior to the launch for personal use, I always found it annoying to either have the stories pop up in a new window, or to load my browser with tabs - the same goes for other RSS readers like Google Reader. I thought of the idea to have the stories open inside the interface, and it seems like something all readers should be doing.

Now to implement it. First, after all of the other HTML elements, two DIV's are created; #feedStory and #feedStoryPanel, both being set to display:none;. #feedStory is empty, and #feedStoryPanel contains all of the links and other objects you see there (on the panel above the embedded page). Each use absolute positioning to fill the screen.

#feedStoryPanel {
  display:none;
  height:85px;
  left:0px;
  position:absolute;
  right:0px;
  top:0px;
  z-index:100;
  background:#FFFFFF url(images/panel-bottom.png) repeat-x scroll center bottom;
  padding-bottom:15px;
}

#feedStory {
  bottom:0px;
  display:none;
  left:0px;
  position:absolute;
  right:0px;
  top:100px;
  z-index:100;
}

To achieve the embedded website, an IFRAME is used. Upon clicking on one of these links, both DIV's are displayed, and an IFRAME (100% width & height) is inserted into #feedStory, with the appropriate address:

$('#feedStory').slideDown("fast");
$('#feedStory').html('<iframe src="' + link + '" width="100%" height="100%" scrolling="yes" frameborder="0"></iframe>');
$('#feedStoryPanel').slideDown("fast");

AJAX (Drupal)

I was astonished to find how easy AJAX-functionality can be implemented with Drupal. Of course, some form of Javascript is needed to facilitate the AJAX request, but I'll explain quickly, the two small steps needed to setup an AJAX function with Drupal.

Step 1: Configuring the URL path with hook_menu

function hook_menu($may_cache) {
  $items[] = array(
    'path' => 'ajaxurl',
    'callback' => 'ajax_function',
    'type' => MENU_CALLBACK,	
  );
}

That registers yourdomain.com/ajaxurl as the target path for the AJAX call. Optionally, you could allow for URL arguments to be used (yourdomain.com/ajaxurl/someargument), to provide a dynamic function.

Step 2: Implement the handler function

As previously specified, we choose ajax_function as our handler function. That means that it will be executed upon visiting ajaxurl. Now the trick with Drupal, is that instead of returning the output data, as you would on any normal Drupal-produced page, you simply print it. Returning data tells Drupal to produce an entire themed page - and we don't want that. We just want the data being produced, so you use print.

function ajax_function() {
    print 'Hello World';
}

Yes, it's that easy.

Aside from using URL arguments to produce dynamic functions, the user object, and regular user access permissions can also be used as they normally are. This seems strange at first because you may think that since jQuery is making the call to the server, that Drupal can't possibly know which user is behind it. Well that simply isn't true, and you can see in the following example:

function ajax_function() {
  global $user;

  if(!$user->uid) {
    print 'You're not logged in!';
  }
  else if(user_access('administer site configuration')) {
    print 'You're an admin!';
  }
  else {
    print 'You're just a regular user!';
  }
}

Streetread only employs three AJAX functions. The first function is for returning the themed feed items. The second function is for receiving the inputted stock symbols. This function verifies the symbols, saves them to your account, and returns a status/error message(s). The last function is for updating the stock symbol sliders after a user changes their symbols.

AJAX Continued (jQuery)

jQuery makes AJAX almost too easy. Before using jQuery, I was using the SACK framework just for AJAX calls, and it was pretty horrible. I'll quickly show how jQuery can be used to work with the previous example of Drupal.

$.ajax({
  type: "POST",
  url: "?q=ajaxurl",
  cache: false,
  success: function(msg){
    $('#main').html(msg);
  },
  error: function() {
    $('#main').html("Error grabbing content.");	
  },
  timeout: 15000,
  dataType: "html"
});

Everything above is pretty much self-explanatory. You can bind AJAX calls like this to any event possible - such as the clicking of a stock symbol:

$('#sliders li').click(executeNews);

#sliders is the div that wraps all of the stock symbols and website icons. Each of those are an LI element, so whenever any of those LI's are clicked, the function executeNews is executed. That function has an AJAX call that is very similar to the example above. The "success" part above is what executes if the AJAX call was made correctly - the "msg" argument is the data received from the call ("Hello World"). It is then printed in the specified div - just how the news items are printed here.

More on jQuery's AJAX here.

Dynamic Content and Event Delegation (jQuery)

This is a quick, and very important, technique that I had some trouble understanding at first. Once a page loads that contains jQuery code, all events are bound to the specified HTML elements (certain links, certain DIV's, etc). When you dynamically load new content into the page, such as after an AJAX call, the new items won't be bound to anything, because jQuery only performs this right after the page loads (hope I explained that correctly). There are some plugins that aim to fix this automatically but I couldn't get any of them to work, and I always prefer to do everything my own way.

In order to bypass this problem, you use a technique called Event Delegation. In simple terms, you tell jQuery to be bound to an object that will always exist, usually in which the dynamic content is located inside. The function that is executed upon the event (like a click) determines if the (clicked) object is what you are trying to bind. I know that sounds confusing - it's hard to explain. Always easy to look at an example..

The DIV #contentContainer is what holds all of the news headlines on this site. The headlines are obviously dynamic, as they change each time you click on a symbol or website logo. You need to bind these headline links to the function that displays the embedded IFRAME, etc. It seems like simply binding '#contentContainer a' would suffice, but as mentioned previously, because these objects keep changing, you can't perform that bind. So instead you bind the entire #contentContainer to the event of a click, and upon a click, you determine if the click was on a link:

$('#contentContainer').click(function(event) {
  if($(event.target).is('a.news-item')) {
    ...
  }
});

To explain, if when #contentContainer is clicked, the target being clicked is a link with the class 'news-item', execute something. Hope that helps.

Other jQuery Plugins & Their Use (jQuery, HTML)

• SerialScroll: Scrolling of stock symbols and website logos

  Not much to say about this plugin, besides that it is extremely easy to use. Create a div with a handful of LI elements that you wish to scroll (just remember to set the width of the UL container to large enough to fit everything & float the LI's left). For the jQuery side of things, just read the simple documentation.

• Tooltip: Tooltips for story descriptions and misc. help

  Again, another very easy plugin to use. Tooltip requires the Dimensions plugin which enables great positioning. Several other tooltip plugins were tested and most of them couldn't handle being located inside a DIV that scrolled. Since the story descriptions are the most complex example using this plugin, I'll explain them.

  HTML:

  //$descID: auto-incrementing number
  //$tooltip: what you hover to see the tooltip 
  <a class="item-desc" href="#" rel="#desc' . $descID . '">' . $tooltip . '</a>
  <div id="desc' . $descID . '" style="display:none;"> (tooltip text here) </div>
  

  jQuery:

  $('#contentContainer a.item-desc').tooltip({
  		delay: 0,
  		showURL: false,
  		bodyHandler: function() {
  			    return $($(this).attr("rel")).html();
  		}
  });
  

  Because of the dynamic content issue explained before, the jQuery code above must be called after each time new headlines are loaded into the interface.

• Thickbox: Overlay for symbol editing

  The use of Thickbox for handling the symbol editing, I believe, was a pretty creative use. Using the Thickbox overlay saves space and enables the entire experience to go without reloading a page. When the page is first loaded, a DIV is created, set to display:none, and filled with the content you see when clicking on the "Edit Stocks" link. To implement Thickbox to show, and to have it display the content of a specific DIV, all you have to do is provide a specific URL:

  #TB_inline?height=400&width=500&inlineId=symbol-edit&modal=true
  

  The inlineId is the ID of the DIV you would like to display. Thickbox also supports AJAX and IFRAME - which is very convenient.

  Using the custom 'Return to Reader' link, instead of the default 'close' button, allows for specific code to be executed upon clicking. In this case, if the symbols have changed, an AJAX call is made to retrieve the users symbols in the HTML format needed to update the symbol slider list.

• asmSelect: Add/Remove/Reorder Website News Sources

  The use of this plugin can be found on registered users "Settings" page (see link in the top menu if you're logged in). The plugin allows users to drag and drop the available websites (the icons on the main page) in the order in which they want them to be displayed. You can also remove or replace sites as you please. Upon submitting, the chosen sites, and the order in which they were sorted, are formatted in a certain way and saved in the database with the users stock symbols. When the main page is displayed, this format string is read, and the sites are displayed accordingly.

  To explain how to handle the displaying and submitting of custom forms in the user account via Drupal, hook_user is used, as the following simple example will demonstrate:

  function hook_user($op, &$edit, &$account, $category = NULL) {
    //Provide form on user settings page
    if($type == 'form' && $category == 'account') {
      $form['address'] = array(
        '#type' => 'textarea',
        '#title' => t('Address'),
        '#default_value' => $edit['address'],
        '#description' => t('What is your address?')
      );
      
      return $form;
    }
  
    //Handle data upon submission
    if($type == 'update' && $category == 'account') {
      db_query("UPDATE {sometable} SET address = '%s' WHERE uid = %d", $edit['address'], $account->uid);
  
      /* IMPORTANT: You must empty out the variable or it will be stored in the user object as well */
      $edit['address'] = '';
    }
  }
  

Aggregation & Cache (Drupal)

Drupal comes with an RSS aggregation module that works quite well. It's used to handle all of the aggregation that powers Streetread. Conveniently, this module has a categorization system in place which is used to separate stock feeds from the site feeds. In the administration panel, my module allows you to specify which category ID is used to hold site feeds. The aggregation module, however, was modified in two ways. Originally, feeds would be set to update after X amount of time. This, I thought, was a bad idea, because in the end, 2500 (or so) feeds would all be updating together, and that could really bog down the server. A configurable option was implemented to update X% of feeds every X minutes. Currently, 25% of the feeds update every 15 minutes - ending up with all feeds updating hourly. And second, the module originally determines feed item deletion based on how old the item is. With popular stocks and sites receiving 30-50 headlines a day, I thought the database would load up way too quick with this system in place. Also, with less popular stocks, there would only be a few headlines left to read. To solve this problem, feeds are given a configurable maximum amount of items, irregardless to age, which will be discussed next with the caching mechanism.

The AJAX-called function to return feed items was first implemented in a very inefficient manner. First the items for the specified source were gathered via a SELECT statement from the database (amongst approximately 100,000 other items). Then, the items (up to 100) were iterated through, adding the needed HTML themeing and other jQuery elements. After all of this, the 40-60kb of data was sent out. I noticed that a single call would send the CPU skyrocketing and only imagined what an active site full of users would do to it.

To solve this problem, a cache was implemented. After each feed is refreshed (via aggregator_refesh) the feed object is sent to a function in my module. The function gathers all of the available items stored in the database for the given feed - in the order of newest to oldest. The first 100 items (or whatever the current setting is), if that many exist, are iterating, themed, and the formatted HTML is stored in a separate table in the database. After this loop, any remaining items are iterated, storing their item ID's in an array, which is then returned to aggregator to remove from the database. Now, the AJAX-handler simply grabs the cached data from the database and returns it.

A user inquired as to how the items are separated by day. A simple technique is used to do so during the previously-mentioned themeing iteration:

  /*
   * $day: variable storing the timestamp of the current day in "z/Y" format; (day of fthe year/year)
   */

  //Print new day if first item 
  if(!$day) {
    $data .= '<div id="feed-day-first" class="feed-day">' . format_date($item->timestamp, 'custom', "l, F jS") . '</div>';
    $day = format_date($item->timestamp, 'custom', "z/Y");
  }

  //Print new day once day changes
  else if($day != format_date($item->timestamp, 'custom', "z/Y")) {
    $data .= '<div class="feed-day">' . format_date($item->timestamp, 'custom', "l, F jS") . '</div>';
    $day = format_date($item->timestamp, 'custom', "z/Y");
  }

Initially, the entire S&P 1500 was preloaded into the database for several reasons, the main being so the most popular stocks would have a large amount of available headlines when users first enter them in. Modules outside of aggregator can dynamically insert feeds into the system via aggregator_save_feed.

Other Drupal Modules & Their Use (Drupal)

Contributed:

• Devel: Provides useful tools and information during development of modules.
• Google Analytics: Makes Google Analytics integration very easy.
• Meta Tags: Provide the home page especially, and nodes with good search engine descriptions and keywords. Great for SEO, even though this isn't exactly a content site.
• Path Redirect: Needed to redirect a certain URL once.
• Pathauto: Provides these blog entries with great, search engine friendly URL's, automatically.
• Token: Required by Pathauto.
• CAPTCHA: Provides those simple math questions on most forms to prevent spam bots from running over this site.
• Views: One of my favorite modules. Provides the blog entries page and block shown while viewing these nodes.

Core:

• Aggregator: (previously mentioned)
• Comment: User comments on the blog entries, for now.
• Contact: The form on the contact page.
• Menu: The menu on the top right
• Path: Required by pathauto.
• Profile: Optional user profiles. Even though there are currently no social features here, these were added.
• Statistics: For my log obsessions.

Feedback & Questions

I hope all of you found this article to be helpful in some way. If you have any feedback, suggestions, or anything at all, please leave a comment below. Important: All questions left below that pertain, and can contribute to, this article, will be answered by appending to the end of the article. This way all helpful questions and answers can be found in one place.

Thanks,
Mike