(Image: Header Graphic)

Tuesday, March 19, 2024

Doug's Domain

Doug Vetter, ATP/CFI

Like what you see?

Donations to dvatp.com are now processed via Stripe. Like this site? It's easier than ever to show your appreciation.

Website Design and Updates

While the primary goal of this site is to provide information I think others would find helpful I also use it as a laboratory to experiment with web technologies.

Major changes to the design (and design philosophy) are documented here for those that care about these sorts of things.

Changes are listed in reverse chronological order.

October 2016

Added Donations Progress Bar

For reasons of privacy I don't disclose who donates to this site or how much they contribute but I have long wanted to provide some feedback about the donations I receive every month.

Since I integrated Stripe payments earlier in the year I've automated the process of collecting donations. As part of that process I save the amount of each donation. The sum total of the donations in the current month is then calculated and used to drive a donations goal progress bar on the front page of the site. The progress bar shows in real time the amount received relative to a modest revenue target I need to pay the hosting bill and continue creating and updating the content that many people reference every day.

So next time you're visiting the home page and you notice the progress bar lagging behind, please donate what you can. When you do, you'll notice the progress bar move forward in direct response to your contribution.

July 2016

Upgraded from Laravel 4 to 5.2

I ported this site to Laravel 4.2 over a year ago and overall I've been happy with my choice. However, Laravel has been under heavy development and while some of the third party packages I have used in conjunction with Laravel still maintain Laravel 4-compatible versions it's clear most development is now focused on Laravel 5. I wanted to gain experience with Laravel 5 for a variety of reasons, including some use cases related to my business, so as my personal site has always served the dual role of pulpit and technology demonstrator it seemed appropriate to migrate the site to Laravel 5.

The biggest issue I see with Laravel 5 so far include Taylor's apparent trimming of the core to exclude basic html, form, input, etc. support. These are now distributed as part of the Laravel Collective effort, so yes -- I had to install what is effectively a third party package to retain the same functionality in Laravel 4. Taylor is, of course, welcome to blaze his own trail and reduce his maintenance burden any way he sees fit, but it does strike me as myopic to exclude this functionality from what he defines as the "core" of Laravel, and worse, not to simply include the necessary packages by default. After all, it's a lot easier to remove functionality than it is to add it -- particularly if you don't necessarily know what you need.

Exception handling was kind of a mess in Laravel 4, and I only recently figured it out. But of course that has completely changed in Laravel 5. The default cases are certainly handled better, which negates needing to learn that much about them out of the gate, but I know I'll need to look into exception handling in greater detail in some of my more exotic applications. In a related note, I can't understand why Taylor removed the so-called pretty whoops page, as I saw that as a distinguishing feature of Laravel. Fortunately there is a way to restore that functionality to Laravel 5 but I haven't done it yet. First things first.

A big part of my port involved namespacing. I have avoided namespacing my PHP code mostly because I could, but as namespacing is thoroughly ingrained in the Laravel 5 development model (after all, the very first artisan command I issued on the fresh install was to set the application namespace) I was forced to update my code. If there is a downside to namespacing it's that use of common Laravel functionality inside of my namespace required me to either prefix all of the classes with a backslash (\) to indicate the "root" namespace (which I find quite ugly) or put one "use" statement at the top of the file for every class I wanted to use. This is reminescent of include and require statements and it resulted in cleaner code overall, so I went with that.

I decided I'd follow up with a feature I had planned since I integrated Stripe payments earlier this year. I have incorporated a simple progress bar on the front page that provides readers with some feedback about the donations I have received relative to a revenue target I need to meet in order to pay the hosting bills.

As as a matter of policy I don't disclose who donates to this site or how much they contribute, but I feel it is important to let you know whether I'm meeting my revenue target every month. Now, when you make a donation and are redirected to the home page the progress bar will advance to reflect your contribution.

June 2016

Moved Donations to Stripe

I have never liked Paypal for a variety of reasons, the chief being they like people to think of them as a bank, yet they are not regulated like one. They can do pretty much anything they want with your money and the reality is they do, often to great harm. The web is replete with stories of how people have been fucked out of their hard-earned money by Paypal for absolutely ridiculous reasons (mostly arcane policy decisions that seem to change with the direction of the wind). Google "paypal sucks". I dare you.

So naturally I have wanted to get away from Paypal and had long ago researched Stripe as an alternative. The snag was that Stripe required TLS and I didn't support that until a couple months ago. With TLS operational, however, it took just a few tedious days of research, development, and (most importantly), testing before I felt sufficiently confident to go live.

I could have taken the easy road and implemented what Stripe calls their "Simple" integration. But that would have forced me to litter my donation page with a multitude of buttons, each specifying a fixed price. But I didn't want to go crazy either and implement an entire form to collect all of the billing information. I compromised by implementing a small form consisting of a drop down menu to select the donation amount as well as a simple button to submit the form and launch the Stripe dialog. I chose the menu over a standard text field mostly to eliminate the need to type an amount which makes the entire process a little less painless.

The great thing about Stripe, as compared to Paypal, is the user experience, and I mean that both in terms of the developer resources provided to people like me who are tasked with its implementation, as well as the experience provided to the end user via the Checkout dialog. Perhaps the best feature of the Checkout from the end-user's perspective is the fact that while the code for the dialog is sourced directly from Stripe's servers the process does not involve any redirects to external sites (*cough* Paypal *cough*). It really is quite slick.

I should point out that in compliance with PCI standards I do not store any payment data on my server. The payment data is encrypted and sent directly to Stripe, which then encodes it in the form of a token. I then store that token temporarily only as required to communicate with Stripe's servers.

April 2016

Added Transport Layer Security (TLS); Disabled Non-Secure Access

I started this site 20 years ago in a different era. Information on the net back then was generally shared freely and openly. Unfortunately, since that time the Internet has become a cesspool of monopolistic ISPs who routinely throttle, filter, and even inappropriately modify data requested by clients to servers they don't control, usually for the purpose of making money above and beyond the ransom they are paid to provide basic Internet access.

Over the last year I've received a few reports from readers concerning some ads they were seeing positioned on various pages. In each case I quickly determined that the ads they were describing were clearly not originating from my server. That meant they were likely being injected from the user's computer via a rogue browser extension or other malware contaminating their system, or possibly the user's local ISP.

I have never wanted to provide TLS support due to reasons of cost and complexity but by late April 2016 I'd had enough. I added TLS support, which encrypts the communications channel between your browser and this server, and permanently disabled insecure access. I further provisioned the server to automatically redirect insecure URLs (beginning with http://) to their secure equivalents (https://).

The good news is as long as you run a modern OS (sorry Windows XP and Vista) and browser that supports SHA256 signatures the effects of this change should go largely unnoticed. The perk of encrypted communications, aside from a notable lack of inappropriate / unauthorized advertising, is that it will limit the ability of your ISP to perform deep packet inspection, site-specific throttling, or other data harvesting. It will also nullify the effect of any transparent proxies or caches so the data you get from dvatp.com will be as fresh as possible.

January 2015

Migration from Codeigniter to Laravel

In prep for some work unrelated to this site I recently learned about Laravel. I found myself intrigued with its featureset to the point that I decided to to port this site to it. The web is replete with articles about Laravel and comparisons to Codeigniter so I'll be brief and just give you the highlights of my experience with it thusfar.

What I like about Laravel:

What I don't like about Laravel:

Database and URI Changes

Porting to Laravel afforded an opportunity to rethink the organizational structure of my code and even my database.

I first refactored my controllers such that they are much more specific to the task and configured Laravel's routing engine to assign a bulk of the URI requests to my rather blandly named ArticleController but more specific parameterized requests to the BlogController or the GalleryController. For example, because order matters in Laravel's routing configuration, I was able to cherry pick /bmw/e36 or /bmw/e46 to go to BlogController while /bmw or /bmw/diy would "fall through" to the ArticleController. Have I mentioned yet how much I like Laravel's routing engine? :-)

I should note at this point that I put my libraries in a custom directory I named "lib" under laravel's app directory. While I could have easily put them in the app/models directory I decided against that for a couple reasons:

  1. They're not models in the strictest sense (i.e. they don't extend the Eloquent-based model base class).
  2. Laravel 5 will remove the app/models directory from the distribution so I'm saving myself a bit of work to relocate the files down the road. Incidentally, I'm not sure I agree with the developer's logic behind that decision but what's done is done.

I've also changed my blog database format and related code such that a blog is now merely a related collection of articles. This allows extensive reuse of the Article library and in the process enables something I've wanted to support for a long time -- navigation of the blog entries without having to go back to the blog index/summary page and support for multi-page blog articles (helpful when trying to present a long blog entry to bandwidth-constrained (e.g. mobile) devices.

As a consequence of this work I made one externally visible change: blog URLs like this:

/bmw/e36/2014/12/01

now mirror the remainder of the site and look like this:

/bmw/e36/2014-12-01

This change has disabled a feature little-used by individual readers and long abused by crawlers. It is no longer possible to create a URI that returns all articles in the given month or year.

Custom Fonts Support

I've known about support for custom fonts in CSS forever but browser support, at least until recently, has been poor. Now that most browsers support custom font downloading I've also updated my CSS to render the site using a couple free fonts: OpenSans for the basic text and Roboto for the headlines. This should eliminate a lot of the stylistic quirks across the various OS platforms. You'll now see the same font I see, regardless of whether you use OSX, Linux, or (gasp) Winders.

I serve these fonts directly from my site largely to ensure quality of service. I know I can get (at least) OpenSans from Google's CDN but I view the very concept of pulling resources from sites I don't control as antithetical to reliability and consistency. I realize the new-age Web2.0 guys are rolling their eyes at that statement but I build fault tolerant embedded systems for a living and the very notion of a distributed web application gives me the heebie jeebies. Bandwidth requirements will increase somewhat but I have enough to burn.

Atom Feed

I turned the home page, which was previously a single static article, into what I call the "news" Blog. This allowed me to dynamically generate the list of articles on the home page as well as an Atom feed you can use with any generic syndication reader, as well as Firefox's built-in "Live Bookmarks".

This feed is presently limited to 750 characters per item so as not to overload your reader. I usually keep the news entries short but if I start ranting the item description will appear truncated in the feed. As I tend to put links at the end of the description for dramatic effect some of these links may therefore not appear in the feed. In that case, simply click on the title of the item and you'll be brought back to the site's news blog for the full text of that news entry.

2013

Migration to HTML5

After painstakingly migrating a bunch of sloppy HTML4.x code to XHMTL 1.0 transitional a few years back I jumped ship from XHTML because it was essentially declared DoA as a web technology about a year ago. Fortunately, because XHTML syntax is strict by design and HTML5 is nearly as strict, as well as compatible with XHTML syntax, I didn't have to do much to convert the site.

The most odd thing about HTML5 I found is that anchors using the "name" parameter are eliminated in favor of the "id" parameter. I discovered that one of my anchor names conflicted with a CSS id used throughout the site so I just changed the anchor name and moved on.

The entire site was tediously sent through the W3C's validator (in beta at this writing) one article at a time and the entire site is now HTML5 compliant. I'm not really using many of the new features, but it does open the door to new experiments.

Core Overhaul

As I've worked with Codeigniter on more personal and professional projects I've developed a better system to organize views. I began with the simple goal to migrate that organizational structure to the site but a few glances at code I hadn't looked at in years was all it took for me to realize it was time for a major overhaul. I began by installing the latest version of Codeignitor, and that required a transition to PHP5. Before long, I had rewritten the core.

There are three externally visible changes resulting from this overhaul.

  1. The content hierarchy has been flattened once again, so all URIs are in the form:

    /section/articlename/articleparams...

    You may notice the new URI format is identical to the typical CI URI (controller/method/params) design I maligned years ago. This change comes from a somewhat belated realization that any relationship between the end-user visible site structure and the URI is coincidental. URIs are now used more to transfer web application parameter data than help orient users in a filesystem-like hierarchy as we have breadcrumbs for that.

    The last remnant of the "URI = Site Structure" philosophy here is the section parameter (bmw, aviation, etc.), and I'm not sure how long that will last. For a brief moment during development I actually considered replacing it with something generic like "/article/" or "/a/" for short but ultimately decided to keep the traditional section names to minimize the number of 301 redirects I had to provision.

    This change also reflects changes to the way people look for and view web data today. No one types URIs manually anymore except to get to the site hostname and most everything is found via search engines (nearly 70% of my visitors come via Google or other search engines), which even further reduces the need to type a hostname. It's not surprising, therefore, that some browser developers are thinking of doing away with the traditional URI bar altogether. I'm not sure I'm ready for that as a user, but as a developer I can see the advantages in hiding my application parameter data from the average end user, particularly if the web developer is given control of the URI bar visibility, i.e. if the server application can tell the client's browser to hide the bar.

  2. I've reworked the top banner to add a search bar which simply points to google (it may point to other less invasive search providers in the near future) and I now utilize some of the nicer features of CSS Version 3 now supported in modern browsers including text-shadow, box-shadow, and transparency to give it greater dimension and visual appeal. All of these things were possible before the advent of CSS3, but with greater effort.

  3. The old CSS-based navigation menu has been replaced with a new version I built more or less from scratch, with inspiration from several online resources. The new menu uses some CSS3 technology for visual appeal and works in conjunction with my core logic to highlight the current section. The markup on which the menu is based is now generated with a menu library I ripped from another open source CMS. It strips me of some flexibility, but precludes me having to maintain the menu markup manually, and that's a good thing.

While some things have changed, one thing remains the same - my use of a flat file database. While most websites have evolved to use SQL based relational databases and I use them professionally, I maintain the same flat file database I created nearly ten years ago. The format remains largely unchanged, and the data preserved. I've never lost any data due to "database corruption", so I'm in no rush to change what works.

2010

In the summer of 2010 I acquired an Apple iPad and I soon realized that certain aspects of the site were difficult to use with a touch interface (for example, page navigation), so I tweaked things as needed.

2007

This year the site's popularity exploded. The average number of unique visitors went up around six times and the monthly bandwidth quintupled. Unfortunately, there was a dark side to this new-found popularity -- the increasing abuse of the site content. For this reason, I decided to rework selected aspects of the backend code again and complete some work I'd neglected to finish in either of the last two major updates.

The results of this update include:

  1. Hotlinking Protection: Images are no longer viewable unless they are visited via one of the site's articles. I used to check the referrer information and give the user a pass if the information was blocked. No more. It really irritates me that I have to take this action, but there are far too many assholes on the net right now. Sometimes I wish it was 1990 again. I'd gladly tolerate the 9600 baud access speeds of the day just to be rid of them.
  2. New CodeIgnitor Controller architecture: Rather than deriving my controllers directly from the CI "Controller" class, I created a "GenericController" class that stores some methods common to all the controllers in use on the site. The new class uses accessor methods (set/get) to manage or override various parameters used to generate the pages. This simple but elegant solution eliminated hundreds of lines of duplicate code and will make it a lot easier to support the site.
  3. New CodeIgnitor View architecture: My views were quite crude, but now they make a lot more sense and are a lot easier to maintain. A "page" view serves only as a template for the page layout with header, content, and footer variables. I added logic that uses a "default" view for each section if I don't otherwise specify it. Those of you who poke at the source of my site will also notice that I now waste valuable bandwidth to explain to people that my content is now available for republication only with written approval. Updating the site to show this text would have been a pain in the ass with my last view architecture, but now it's in a single place and easy to update.
  4. Reorganized and improved CSS: Rather than serving one huge CSS file for each page of the site, I cut up the CSS and modularized it so I only send the CSS I need to render a given page.
  5. Added CSS-based hierarchical site menu (look Ma, no Javascript!) based on ADxMenu. I developed something similar to this back in 2003 following Eric Meyer's lead but never published it because I couldn't get certain aspects of the menu to render properly in all browsers. Rather than sink more time into my own project, I decided to try ADxMenu. Looks good so far.

2006

Philosophy

The simplest websites (like mine) traditionally serve data from a directory structure that rigidly defines the structure of the website as viewed by the user. In other words, the user sees what the developer sees. If you browsed to "/bmw/articles/tools/" on this site, for example, the server would traverse a matching directory tree and serve the data in the "tools" directory. This served its function well but it had recently become irritating and limiting from a developer's perspective for a number of reasons.

The solution to the problem was to pitch the outdated core I'd built many years ago before I really knew PHP or how to build websites effectively and replace it with a new PHP-based framework. I'd contemplated building my own, but only foolish developers reinvent the wheel, so I went on a quest for the ultimate prefabricated framework. I had problems finding a framework that would be simple enough to learn quickly, mesh well with my static website and its flat-file contents, and implement the many good practices associated with object oriented design. I ultimately narrowed it down to CakePHP and CodeIgnitor but wound up discarding CakePHP due to complexity and the fact that its primary benefit over other packages was its scaffolding features -- something I wouldn't use.

As I began to survey CodeIgnitor in detail, I found I liked the thorough documentation, PHP 4/5 support, clean interfaces, extensibility, active development schedule with releases every few months and a notable lack of attitude on the part of its developers. Like most frameworks, it supported the so-called MVC (Model / View / Controller) development model that strives to separate the data from the presentation with a intermediary or decision-maker (controller) to translate the raw data from the database into a form acceptable to the user. This MVC format usually extends to the user in the form of a URI in the form "/class/method/parameter1/parameter2/...".

It became apparent that certain aspects of the hierarchial nature of my site's original design were preferable to the traditional MVC URI format, but I also did some reading and stumbled upon the so-called three-click rule. Right or wrong, it aims to point out that users become frustrated if they're required to click more than three times from the front of a website to find the data they need. This forced me to rethink the structure of my site. The end result is a slightly "flatter" structure that's easier for the user to navigate and easier fo the developer to maintain.

One of the other challenges of the MVC URI is building a breadcrumb history such as "Home->Aviation->Reviews". After experimenting for many hours with other people's breadcrumb classes, including one specifically designed to work with CodeIgnitor and the URIs that result from use of Apache's mod_rewrite module, I decided to dump all the complexity and write my own, The result is so stupidly simple I have no idea why I haven't seen an example like it elsewhere. The proof is in the pudding. The breadcrumbs exist, but the backend organization of the data bears no resemblance to the hierarchial structure implied by the breadcrumb list.

I was also motivated to make changes to the organizational structure of the site so I could better serve images and other types of media and do it in a way that would not be abused by people who like to hotlink or embed my content in their pages in defiance of my site's terms of service. While much of this work is not yet complete, the framework to support the work is now in place.

Change List

The biggest changes occurred on the backend of the site and are not visible to users. However, while I was "under the hood" so to speak, I took the time to give the site a facelift.

First of all, all image and many article URI's have changed as a consequence of the redesign.

The MVC format and coding style mandated one major change in the organizational structure of the BMW section. Rather than the URI being "/bmw/<type>/maintenance", it's now "bmw/maintenance/<type>". I did this to simplify my backend programming, as "bmw" is the controller name, "maintenance" is the name of a method (function), and everything that follows it (including the vehicle type) is a parameter. The biggest benefit to this structure is cleaner URIs (anchors are mostly eliminated, particularly in the BMW maintenance logs), and the fact that users can now directly address and bookmark a single maintenance log entry by date. The perk for me is that my bandwidth should drop because you won't be forced to download the year's worth of log entries when all you want to see is the latest, and I'll be able to better track what log entries you find helpful. That will allow me to direct updates to the most needed sections.

Again, to simplify the backend development, I flattened the Aviation section structure. The "articles" portion of the URI was removed and the articles subsections brought to the aviation index page. Reviews are now located at "/aviation/reviews/" rather than the more verbose "/aviation/articles/reviews". A small change for the user, but a helpful one I think. Think "three click rule".

I'm now benchmarking each page. In the very bottom right corner of each page you'll find a rendering time. That's the time the server required to build the page and send it out the interface. This is a general indication of my coding efficiency but it can also indicate an overloaded server. Before you report that my site appears slow, check the rendering time. If it's less than a second (it will usually be far less than that), the problem is out of my control.

A lot of cruft had formed in various parts of the site over the past two years. That's been eliminated and replaced with proper XHTML and CSS constructs. I'm still serving pages as XHTML 1.0 Transitional with a type of "text/html". Yea, I know that's not correct, and yea, I know the W3C is now backpedaling on XHTML in favor of extensions to good old HTML 4.01. I would have made the necessary changes, but I had different priorities this time around. Perhaps next time. Or, perhaps by next time Micro$oft will get their act together and support proper rendering of "application/xml".

I deleted a couple of articles that were out of date and/or worthless at this point.

2004

State of the Art - XHTML and CSS2

This iteration of the site was fueled by my desire to meet current open standards in order to promote accessibility and consistent presentation. This has become possible with XHTML and CSS combined with recent development in browser technology in support of these standards. This presented the most radial cosmetic redesign and resulted in the red, white and blue theme you see to this day.

Single Vs. Two/Three Column Layout

Let's face it. The typical two or three column web page template was designed to position ads and drive the content per page down so more pages get clicked through to increase ad rotation. Since I don't do traditional advertising to support this site, I made it a point to create a single column page design with navigation links at the top and bottom of the page. This allows me to include large thumbnail images with the content (you remember, the stuff on a website that people want to read), and keeps the optimal browser width down to a minimum -- thus saving you precious desktop space.

Original Designs (1995-2004)

Static HTML, then a misguided dabbling with frames

The initial site used a few static html pages in a single directory. As the site grew, I got tired of maintaining the navigation links on each page, so I transitioned the site to use frames. In retrospect, frames were a crutch. I wanted a menu on the site but didn't want to put in the time necessary to build a site whose pages were dynamically generated. And who could fault me. PHP didn't exist in any substantial form at the time, and I wasn't about to punish myself with the perl of the day.

Frames worked pretty well for a few years and allowed me to focus on content development. During this time, I became more familiar with various web standards and began to transition the site to HTML 4.01.

While I might say that I was motiviated to begin using Cascading Style Sheets at this time because I wanted to begin to separate markup from presentation, in reality I hated the standard look of hyperlinks and wanted an easy way to change their attributes. Psuedo-classes provided by CSS Version 2 provided that capability. At this time, however, I did not use CSS2 for positioning of content on the page (which is ironic, since that's the primary billing of CSS!) because browser support was flaky at best and the dreaded Netscape 4.x was still the dominant browser.

An experiment with Templates

While researching PHP I happened upon an article that described the use and benefits of template engines, and the FastTemplate class in particular. Many of the concepts appealed to the object-oriented designer in me, so I then decided to eliminate frame support and replace it with the FastTemplate class.

I divided the page into three simple logical blocks -- a header, content, and footer. Aside from a slightly different footer template for the front page, all pages used this format and made it easy to make global changes to the header and footer (meaning, the site menu, copyright info, etc.).

The bliss was relatively short lived, however. When my provider upgraded the version of PHP on my webhost, the FastTemplate class balked at a few of the changes and I had to fix them. That forced me to learn more about PHP than I wanted to at the time, but in retrospect this set the stage for future work. I continued using the class for another year or so, but since the class wasn't (IMHO) particularly well written and the author was no longer developing it to keep pace with the updates to PHP, I began a search for other solutions.

I quickly found the Smarty template system and in fact almost used it in the latest iteration, because it's pretty much the top-shelf in templating systems, but I ultimately decided to eliminate templates in favor of raw PHP. I determined that unless a site uses a backend database and has lots of associated logic, it's just simpler and faster to use raw PHP to divide the page into blocks and create a simple template for the site.

In other words, as long as the code is developed with care to separate logic from presentation and you don't need the data conversion utilities provided by the template system, PHP is a fine template system in its own right.