Technologies To Learn

InnoDB plugin row format performance

2012-01-21T02:23:00.001-08:00

InnoDB plugin row format performance

Here is a quick comparison of the new InnoDB plugin performance between different compression, row formats that is introduced recently.

The table is a pretty simple one:

CREATE TABLE `sbtest` (
`id` int(10) unsigned NOT NULL,
`k` int(10) unsigned NOT NULL DEFAULT '0',
`c` char(120) NOT NULL DEFAULT '',
`pad` char(60) NOT NULL DEFAULT '',
PRIMARY KEY (`id`),
KEY `k` (`k`)
) ENGINE=InnoDB ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8;

The table is populated with 10M rows with average row length being 224 bytes. The tests are performed for Compact, Dynamic and Compressed (8K and 4K) row formats using MySQL-5.1.24 with InnoDB plugin-1.0.0-5.1 on Dell PE2950 1x Xeon quad core with 16G RAM, RAID-10 with RHEL-4 64-bit.

Here are the four test scenarios:

1. No compression, ROW_FORMAT=Compact
2. ROW_FORMAT=Compressed with KEY_BLOCK_SIZE=8
3. ROW_FORMAT=Compressed with KEY_BLOCK_SIZE=4
4. ROW_FORMAT=Dynamic

All the above tests are repeated with innodb_buffer_pool_size=6G and 512M to make sure one fits everything in memory and another one overflows. The rest of the InnoDB settings are all default except that innodb_thread_concurrency=32.

Here is the summary of the test results:

Table Load:

Load time from a dump of SQL script having 10M rows (not batched)
Compact Compressed (8K) Compressed (4K) Dynamic
28m 18s 29m 46s 36m 43s 27m 55s

File Sizes:

Here is the size of the .ibd file after each data load
Compact Compressed (8K) Compressed (4K) Dynamic
2.3G 1.2G 592M 2.3G

Data and Index Size from Table Status:

Here is the Data and Index size in bytes from SHOW TABLE STATUS and you can see the original data size here rather than the compressed size
Compact Compressed (8K) Compressed (4K) Dynamic
Data 2247098368 2247098368 2249195520 2247098368
Index 137019392 137035776 160301056 137019392

Compression Stats:

Here is the compression stats after the table is populated from information_schema.InnoDB_cmp; and you notice that 4K takes more operations and time for both compression and un-compression
Page_size Compress_ops Compress_ops_ok Compress_time Uncompress_ops Uncompress_time
8K 8192 446198 445598 73 300 0
4K 4096 1091421 1012917 463 38801 13

Performance:

Here is the performance of various row formats with threads ranging from 1-512 for both 512M and 6G buffer pool size for both concurrent reads and writes.

compress512m

compress6g

Observations:

Few key observations from the performance tests that I performed without looking to any of the sources, as I could be wrong, someone can correct me here. Its hard to draw from these input scenarios, but helps to estimate what is what.

* The load time is almost same except that the 4K compression seems to take longer than the rest; and compression in general is hitting the INSERT/Load performance a little bit.
* Compact or Dynamic, there is no compression; so the data and index file sizes will be almost same
* The SHOW TABLE STATUS for compressed table will have its original Data_Length and Index_Length statistics rather than the compressed statistics (may be a bug or InnoDB needs to extend SHOW TABLE STATUS to show any compressed sizes or other means, right now only option is to view your files manually)
* 8K compression reduced the .ibd file by nearly 50% (1.2G out of 2.3G) and 4K compression reduced the size by 1/4th (592M out of 2.3G); and it could vary based on table types and data.
* 8K compression takes less ops and time for both compression and de-compression when compared to 4K (obvious)
* When there is enough Innodb buffer pool size to act data in memory, the compression is a bit overhead, but you will be saving space
* When there is a overflow from buffer pool (IO bound), compression seems to really help
* 4K compression in general seems to be slower when compared with 8K or any other row_format.

How To Obtain hierarchical data / Parent - Child relationship

2012-01-21T01:55:00.000-08:00

Explain How To Obtain hierarchical data / Parent - Child relationship data

This is a series of articles on hierarchical queries in MySQL:

IPv4 vs IPv6

2012-01-21T00:53:00.000-08:00

I had compiled differences between IPv6 and IPv4 long back.

Hope someone might find this useful.

IPv4	IPv6
Addresses are 32 bits (4 bytes) in length.	Addresses are 128 bits (16 bytes) in length
Address (A) resource records in DNS to map host names to IPv4 addresses.	Address (AAAA) resource records in DNS to map host names to IPv6 addresses.
Pointer (PTR) resource records in the IN-ADDR.ARPA DNS domain to map IPv4 addresses to host names.	Pointer (PTR) resource records in the IP6.ARPA DNS domain to map IPv6 addresses to host names.
IPSec is optional and should be supported externally	IPSec support is not optional
Header does not identify packet flow for QoS handling by routers	Header contains Flow Label field, which Identifies packet flow for QoS handling by router.
Both routers and the sending host fragment packets.	Routers do not support packet fragmentation. Sending host fragments packets
Header includes a checksum.	Header does not include a checksum.
Header includes options.	Optional data is supported as extension headers.
ARP uses broadcast ARP request to resolve IP to MAC/Hardware address.	Multicast Neighbor Solicitation messages resolve IP addresses to MAC addresses.
Internet Group Management Protocol (IGMP) manages membership in local subnet groups.	Multicast Listener Discovery (MLD) messages manage membership in local subnet groups.
Broadcast addresses are used to send traffic to all nodes on a subnet.	IPv6 uses a link-local scope all-nodes multicast address.
Configured either manually or through DHCP.	Does not require manual configuration or DHCP.
Must support a 576-byte packet size (possibly fragmented).	Must support a 1280-byte packet size (without fragmentation).

Network Sorcery is a great place to find RFC(s).

Refer to http://www.networksorcery.com/enp/protocol/ipv6.htm and http://www.networksorcery.com/enp/protocol/ip.htm links for related RFC(s) of IPv6 and IPv4 respectively.

Also there is good reference for Understanding IPv6 @ http://technet.microsoft.com/en-us/library/cc786127.aspx

ET:QuakeWars - Linux Client is now available

2007-10-25T00:39:00.000-07:00

In case you missed the official announcement, ETQW Linux Client is now available to download. You do need the full retail DVD to complete the installation though. The client itself is really small at 18MB, so need to work out if they actually preloaded some OS compatilibty stuff on the DVD pre-release.

Another issue, it only runs in 32bit mode, so if you have a x86_64 install - you will need the compat libs along with their deps. My main gaming machine is i686, but my primary laptop is x86_64 ( the Acer Ferrari 4005 ) and I will at some point install the game there as well. So a list of rpms required on centos-5 to make it work will get posted here soon'ish :)

The Oracle Unbreakable Linux incompatibility

2007-10-25T00:32:00.000-07:00

Submitted by dag

It was exciting when a year ago at LinuxWorld Expo London we heard a rumour that Oracle was going to support Red Hat Enterprise Linux. It was surprising to find out that this rumour was not entirely as well-intended and supportive as it sounded.

Instead of helping the community, Oracle was directly attacking Red Hat with its own product as if it was showing the world that Open Source has no value and that even Oracle could run away with Red Hat's crown jewels and customers. A hostile take-over attempt of Red Hat by Oracle.

What the world did not understand (and Oracle tried to hide) is that the business value is not in the source-code, but in the development and the community. Red Hat is involved in most of the Open Source projects and in fact is able to support their product for 7 years because they have the experts and the insight of how projects work. Oracle has not.

Another well-hidden fact of Oracle's promotional buzz is that you cannot both be compatible with RHEL, and provide bugfixes and improvements. Either you make changes, or you stay compatible with the original. So whatever Oracle stated was self-contradictory. All the articles at the time failed to mention that, riding on Oracle's wave.

Now that Oracle is porting YaST to Red Hat (who in their right mind would want to have YaST on RHEL is beyond my comprehension) it becomes all that obvious that whatever procedure you have for RHEL or CentOS it will not work with YaST. Once you use YaST, you cannot change (a lot of) configuration files by hand.

I always tell people at tradeshows that "CentOS is completely compatible with Red Hat Enterprise Linux, including the bugs". And even though that seems a bit harsh, it is very true and it is exactly what we want to tell people. We cannot replace Red Hat's support because we need to be compatible and therefor rely on Red Hat's support. We cannot fix any bugs ourselves. That means that if you really need support (and you cannot support yourself), you need to get it from Red Hat.

The CentOS users rely on us to provide a compatible product and Oracle is fooling their customers if they tell them otherwise...

A Visual Explanation of SQL Joins

2007-10-24T03:54:00.001-07:00

I thought Ligaya Turmelle's post on SQL joins was a great primer for novice developers. Since SQL joins appear to be set-based, the use of Venn diagrams to explain them seems, at first blush, to be a natural fit. However, like the commenters to her post, I found that the Venn diagrams didn't quite match the SQL join syntax reality in my testing.
I love the concept, though, so let's see if we can make it work. Assume we have the following two tables. Table A is on the left, and Table B is on the right. We'll populate them with four records each.

id name       id  name
-- ----       --  ----
1  Pirate     1   Rutabaga
2  Monkey     2   Pirate
3  Ninja      3   Darth Vader
4  Spaghetti  4   Ninja

Let's join these tables by the name field in a few different ways and see if we can get a conceptual match to those nifty Venn diagrams.

SELECT * FROM TableA
INNER JOIN TableB
ON TableA.name = TableB.name

id  name       id   name
--  ----       --   ----
1   Pirate     2    Pirate
3   Ninja      4    Ninja

Inner join produces only the set of records that match in both Table A and Table B.

SELECT * FROM TableA
FULL OUTER JOIN TableB
ON TableA.name = TableB.name

id    name       id    name
--    ----       --    ----
1     Pirate     2     Pirate
2     Monkey     null  null
3     Ninja      4     Ninja
4     Spaghetti  null  null
null  null       1     Rutabaga      
null  null       3     Darth Vader

Full outer join produces the set of all records in Table A and Table B, with matching records from both sides where available. If there is no match, the missing side will contain null.

SELECT * FROM TableA
LEFT OUTER JOIN TableB
ON TableA.name = TableB.name

id  name       id    name
--  ----       --    ----
1   Pirate     2     Pirate
2   Monkey     null  null
3   Ninja      4     Ninja
4   Spaghetti  null  null

Left outer join produces a complete set of records from Table A, with the matching records (where available) in Table B. If there is no match, the right side will contain null.

SELECT * FROM TableA
LEFT OUTER JOIN TableB
ON TableA.name = TableB.name
WHERE TableB.id IS null

id  name       id     name
--  ----       --     ----
2   Monkey     null   null
4   Spaghetti  null   null

To produce the set of records only in Table A, but not in Table B, we perform the same left outer join, then exclude the records we don't want from the right side via a where clause.

SELECT * FROM TableA
FULL OUTER JOIN TableB
ON TableA.name = TableB.name
WHERE TableA.id IS null
OR TableB.id IS null

id    name       id    name
--    ----       --    ----
2     Monkey     null  null
4     Spaghetti  null  null
null  null       1     Rutabaga
null  null       3     Darth Vader

To produce the set of records unique to Table A and Table B, we perform the same full outer join, then exclude the records we don't want from both sides via a where clause.

There's also a cartesian product or cross join, which as far as I can tell, can't be expressed as a Venn diagram:

SELECT * FROM TableA
CROSS JOIN TableB

This joins "everything to everything", resulting in 4 x 4 = 16 rows, far more than we had in the original sets. If you do the math, you can see why this is a very dangerous join to run against large tables.

Best Places To Use Ajax

2007-10-24T03:53:00.001-07:00

Form driven interaction.
Forms are slow. Very slow. Editing a tag (the old way) on a del.icio.us bookmark? Click on the edit link to load the edit bookmark form page, then edit the field and hit submit to wait for the submission to go through, then return to the previous page and scroll down to find the bookmark to see if the tags look right. Ajax? Click on the edit link to instantly start changing tags, click on the submit button to asynchronously send off changes to the tags and quickly see in place what changed, no reloading the entire page.

Form driven interaction- Subset:Linked Select Menus. Imagine a T-Shirt with 3 options; Size, Color, and Style. When tracking inventory for your product, you know you have Large, Red, Polo shirts in stock, but you’re out of Small, Blue, T-Shirts… It is frustrating to the user to pick this combination and then receive an error on the checkout page stateing that you are out of stock… and then have to go back to the selection process and reconfigure the item… Using AJAX, you can check the stock of the options as the user picks them and only return or show the items which are in stock.
Form driven interaction- Subset: Autosave. Think of someone writing in Word. Which button do they use the most? Save.
With javascript you can do one better. Not only can you have a save & continue that works just like the del.icio.us forms – you can autosave! Remember to tell the user this, as simply knowing this relaxes quite a lot of people. Properly explained count-down clocks are prefered, for obvious reasons.

Deep hierarchical tree navigation. First of all, applications with deep hierarchical tree navigation are generally a nightmare. Simple flat topologies and search/tagging works very well in most circumstances. But if an application really calls for it, use Javascript to manage the topology ui, and Ajax to lessen the burden on the server by lazy loading deep hierarchy data. For example: it’s way too time consuming to read discussion threads by clicking through and loading completely new pages to see a one line response.

Rapid user-to-user communication. In a message posting application that creates immediate discussions between people, what really sucks is forcing the user to refresh the page over and over to see a reply. Replies should be instant, users shouldn’t have to obsessively refresh. Even Gmail, which improves on the old hotmail/yahoo mail ‘refresh inbox, refresh inbox’ symptom, doesn’t really push Ajax far enough yet in terms of notifying new mail instantly.

Voting, Yes/No boxes, Ratings submissions. It’s really too bad there are no consistent UI cues for Ajax submission, because submitting a vote or a yes/no response is so much less painful when the submission is handled through Ajax. By reducing the time and impact of clicking on things, Ajax applications become a lot more interactive – if it takes a 40 seconds to register a vote, most people would probably pass unless they really care. If it takes 1 second to vote, a much larger percentage of people are likely to vote.

Filtering and involved data manipulation. Applying a filter, sorting by date, sorting by date and name, toggling on and off filters, etc. Any highly interactive data manipulation should really be done in Javascript instead of through a series of server requests. Finding and manipulating a lot of data is hard enough without waiting 30 seconds between each change in views, Ajax can really speed this up.

Commonly entered text hints/autocompletion. Entering the same text phrases or predictable text phrases is something software/javascript can be good at helping out with. It’s very useful in del.icio.us and GMail, for quickly adding tags/email addresses.

Interactive Errors If someone is entering complicated data, it doesn’t make sense to tell them they have failed only after a lengthy submission process. Ajax can speed up this workflow by quickly letting the user know of an error condition before they try to submit. Example: a username chooser, instead of making the user submit the entire form, try a new name and repeat, or keep trying a ‘is this name chosen’ form, the username chooser can simply indicate to the user whether the username is unique or not, while the user is still typing it.

Long Running Queries/Remote Calls If a query or a call to a remote webservice is going to take a long time that cannot be avoided, Ajax works well to manage the time a user waits for the call to return. For example, SWiK uses Ajax to fill in results from webservices detailing new projects: a user doesn’t have to wait for Google webservice to return before starting to edit a new project

Computationally Expensive Operations Unfortunately, Javascript has a tendency to be quite slow. Complex math or number crunching just isn’t Javascript’s forte. Additionally, heavy Javascript computation can slow the basic user interface to a crawl. An XMLHTTPRequest call can be helpful here, pushing expensive computations to beefier remote servers.

Server Savings

Sometimes, a process users do over and over on a site requires only a small amount of new data to be sent over the wire, but loading entire new pages can be a strain on the servers in bandwidth and resources. Ajax can be used to load pages more efficiently, as seen in various tests. Of course the ease of making new or multiple requests from the server using Ajax also means that it’s easy to overtax server resources as well.

Interactive Panning And Moving Over Data

Moving and scanning over large data sets makes it impracticable to pre-load all of the data. Loading the data just ahead an just behind the user gives the appearance of the entire data set being accessible, and helps eliminate loading times. A great example of this is Google Maps’ scrolling tiles system that gives the effect of moving over a map by picking up tiles behind and placing them ahead of the user, filling them with new data requested via Ajax.

Best Top Ten Open Source,Ajax/DHTML Librearies For Web Developer

2007-10-24T03:52:00.002-07:00

Hi, Frnds...I made a list of the top 10 libraries that I have come across or that I personally use. Libraries can be best for a web developers friend. They are great resources to learn from and can save hours and hours of time. These libraries include JavaScript, Ajax, Colors, PHP, and CSS. These should be in any web developers bookmarks, so go ahead and look through these libraries and bookmark your favorite ones. The list is in no particular order.

1) Moo.fx - A superlightweight, ultratiny, megasmall javascript effects library, written with prototype.js. It’s easy to use, fast, cross-browser, standards compliant, provides controls to modify Height, Width, and Opacity with builtin checks that won’t let a user break the effect with multiple crazy clicks. It’s also optimized to make you write the lesser code possible.

2) Rico - An open source JavaScript library for creating rich internet applications. Provides full Ajax support, drag and drop management, and a cinematic effects library.

3) Swat - Developed by silverorange, Swat is an open source web application toolkit built with PHP.

4) ColorCombos - Who would’ve thought a color library would end up mixed in with a bunch of JavaScript and PHP libraries? Well they do have a pretty sweet little color library for finding color combinations, all you do is select the color and they show you some nice combos that work with that color.

5) script.aculo.us - Provides you with easy-to-use, compatible and, ultimately, totally cool JavaScript libraries to make your web sites and web applications fly, Web 2.0 style. I’m sure I’m not alone when I say this library is my favorite.

6) Mochikit - A kick-ass lightweight JavaScript library that will help you get shit done fast.

7) Dynamic Drive CSS Library - Here you’ll find original, practical CSS codes and examples such as CSS menus to give your site a visual boast.

8) PEAR - A framework and distribution system for reusable PHP components. PEAR provides the above mentioned PHP components in the form of so called “Packages”.

9) DHTML Goodies - A good sized library of DHTML and AJAX scripts.

10) dojo - Open source JavaScript toolkit that makes professional web development better, easier, and faster.

Honorable Mentions

11) Cross Browser | Toys - Huge JavaScript library.

12) Yahoo UI Library - The Yahoo! User Interface (YUI) Library is a set of utilities and controls, written in JavaScript, for building richly interactive web applications using techniques such as DOM scripting, DHTML and AJAX. The YUI Library also includes several core CSS resources.

Big thanks to all of those who have help in anyway to put one of these libraries together.

I hope you find this list helpful. Keep in mind there’s hundred of libraries available online,

Income Streams for Bloggers

2007-10-24T03:52:00.001-07:00

Advertising Programs - Perhaps the most obvious changes in the past few months have been with the addition of a variety of viable advertising options for bloggers. No longer are bloggers only presented with the Adsense and/or BlogAds choice - instead they now have a massive array to choose from. Getting the most publicity recently have been Chitika’s eMiniMalls of course but there are just so many other options now that also include:

Adgenta, CrispAds, Text Link Ads, Intelli Txt, Peak Click, DoubleClick Tribal Fusion, Adbrite, Clicksor, Industry Brains, AdHearUs, Kanoodle, AVN, Pheedo, Adknowledge, YesAdvertising, RevenuePilot TextAds, SearchFeed, Target Point, Bidvertiser, Fastclick Value Click and OneMonkey (to name just some of the options - I’m sure I’ve forgotten some) and there is a smorgasbord of options. Of course there is more to come with MSN Adcenter and YPN both in beta testing and with a variety of other advertising system currently in development (so I hear).

RSS Advertising - The past 12 months have seen some advances in RSS Advertising also. I’m yet to hear of any bloggers making big dollars through it to this point - but as improvements are made to the ad programs exploring this I’m sure we’ll start to see examples of it being profitable.

Sponsorship - In addition to the array of advertising programs that are available to join there is a growing awareness in the business of the value and opportunity that exists for them to advertise directly on blogs. I’m hearing more and more examples of this and have been fortunately to have a couple of ad campaigns of my own in the past month - one with Adobe a couple of weeks ago and another just completed with Ricoh for a new digicam over at my Digital Camera Blog. These are not isolated cases - as I say I know of many blogs exploring sponsorship with advertisers at present and suspect we’ll see more of it in the year ahead. Sponsorship is also happening on a post by post basis with some bloggers being paid to write on certain topics by companies - either in one off or a regular fashion.

Affiliate Programs - There are larger affiliate programs like Amazon, Linkshare, Clickbank and Commission Junction but also literally thousands of others from the large to the very small.

Blog Network Opportunities - with the rise in popularity of Blog Networks - bloggers are also being presented with more places to earn an income from their blogging - by writing for and with others. While it might be difficult to get a writing gig with one of the bigger networks - there are plenty who are always asking for new bloggers to join and who are willing to pay bloggers using a variety of payment models. While there are distinct advantages of blogging for yourself - blogging for an established network who will handle a lot of the set up/promotion/admin/SEO etc has it’s advantages also. More and more bloggers are combining writing for themselves on their own blogs with taking on blog network blogs as additional income streams.

Business Blog Writing Opportunities - as blogging has risen in it’s profile as a medium more and more businesses are starting blogs. Many of these companies have internal staff take on blogging duties - but an increasing number of them are hiring specialist bloggers to come on and run their blogs. I know of a number of bloggers who in the past month or two have been approached for such paid work. Check out Bloggers for Hire if you’re looking for this type of work.

Non Blogging Writing Opportunities - Also becoming more common are bloggers being hired to write in non blogging mediums. Manolo’s recent coup of a column in the Washington Post is just one example of this as bloggers are increasingly being approached to write for newspapers, magazines and other non blog websites. Along side this is the rise of bloggers as published book authors - this is to the extent that one blogger I spoke with this week complained to me that they were one of the few bloggers than they knew who didn’t have a book deal!

Donations - Tip Jars and donation buttons have been a part of blogging for years now but this last year saw a number of bloggers go full time after fundraising drives. Perhaps the most high profile of these was Jason Kottke of kottke.org who through the generosity of his readership was able to quit his job and become a full time blogger.

Flipping Blogs - Also more common in 2005 was the practice of ‘Blog Flipping’ - or selling of blogs. This has happened both on an individual blog level (I can think of about 20 blogs that sold this year) but also on a network level (the most obvious of these being the 8 figure sale of Weblogs Inc to AOL).

Merchandising - My recent attempt to sell ProBlogger.net T-shirts wasn’t a raging success, but it is an example of how an increasing number of bloggers are attempting to make a few extra dollars from their blogs by selling branded products through programs like Cafepress (although I have to say they’ve lost one of my own orders and are being quite unresponsive to my requests to follow it up at present). While I didn’t have a lot of success with merchandising - quite a few larger blogs are seeing significant sales - especially blogs with a cult following. I’m not at liberty to discuss details - but I know of one largish blog which will see sales over $20,000 in merchandise for the calendar year of 2005.

Consulting and Speaking - While it has been popular for established consultants to add blogs to their businesses we’re also starting to see bloggers with no consulting background earning money by charging readers for their time in consulting scenarios BECAUSE of the profile that their blogs have built them. Blogging has the ability to establish people as experts on niche topics and we all know the value of being perceived as an expert. I spoke to one blogger last month who charges himself out at over $200 an hour for speaking and consulting work - his area of expertise was something that he knew little about 18 months ago - but through his blog he’s become a leader in his field and a minor celebrity in his industry.

As time rolls on there are more and more blog earning opportunities opening up. Feel free to suggest your own ideas in comments below.

Open source databases is "SIXTY%" cheaper !!!!!

2007-10-24T03:51:00.000-07:00

You know .....that Open source databases can save enterprises up to 60 per cent over proprietary products, according to data collected by
recent searches.

A senior analyst at famous database management systems, estimated that average savings on the total cost of ownership are about 50 per cent. The data is based on surveys and customer interviews.

Open source databases such as Enterprise DB, Ingres and MySQL do not carry licence fees, and management tools. Soit is less expensive than for proprietary databases from Oracle, Microsoft and IBM.

Open source offers especially their proprietary competitors in low-end applications with databases of less than 200GB in size outshininigly .

The one fact os this research is that "Eighty per cent of the applications typically use only 30 per cent of the features found in commercial databases," and "The open source databases deliver those features today."

But the hitch is that open source databases generally lack the features for mission critical applications, trailing behind their proprietary peers in security, uptime, performance and features such as XML support.

Enterprise applications from Oracle and SAP also do not support open source databases today, but right now condition expects that to change "within a couple of years".

Open source database vendors typically do not position their products as low-cost alternatives.

But customers still consider price as the primary benefit of open source, ya this is fact.

"The number one reason why any customer would choose an open source database is cost. That still holds true today" .

But the low price is also enabling companies to set up new projects that would previously have been too expensive, such as data mining of log files and setting up data repositories.

In an attempt to the competition from low-cost open source databases, Oracle launched a free database last year that is essentially a scaled down version of its enterprise grade Oracle Database 10g.

The application targets test deployments for developers and students rather than enterprises.

10 Valuable Tips for Creating Your Web Site

2007-10-24T03:50:00.001-07:00

Introduction

When looking for ways to build of your web site, even minor steps can make a huge difference. The most helpful information and best content will have little impact without simple protocols that make your Web site easier to use and more visually appealing. This paper focuses on 10 tips you can employ to ensure your web site is effec- tive from the day it goes live.

1. Accessibility

Web site accessibility has recently become a very important issue in the web community. Because of Section 508 of the Rehabilitation Act of 1973, all web sites and pages created by Federal agencies and Federal contrac-tors after June 21, 2001 must comply with its provisions. The purpose of the law is to make web sites accessi-ble to all individuals, including those with disabilities. The World Wide Web Consortium (W3C) developed Web
Content Accessibility Guidelines (May 1999), which expand the scope of Section 508.

This is very important to all web developers, whether you are a government agency, a contractor who does work for the government, or a private firm with its own web site. An accessible web site refers to any content or information provided via an online medium that all individuals (including those with disabilities) could easily access and understand. Disabilities include not only visual impairments, but auditory, cognitive, and physical
impairments as well. They can range from very severe (total blindness, for example) to something as simple as the increasing inability to see contrasts that develop as we age. Greater accessibility means more people can fully utilize your web site’s features.

There are a number of aids available for enhancing accessibility. These range from programs like Jaws or IBM’s Home Page Reader—which read the page aloud for the visually impaired—to sip-and-puff systems for the quadriplegic. It is up to the web developer to create pages that allow these systems to provide an equivalent alternative for these individuals. Think of accessing a web page as being similar to a play or opera. If you read the script without benefit of the actors’ interpretation, lights, scenery, or music, you are only experiencing one aspect of the author’s intent. The same is true of web access: if you can only hear the words being read, for example, with no description of the images on the page or other visual components, you would not fully expe-rience the information being provided on the page. That is why, when we include an image, it is important to use the “alt” attribute to provide a description of the image. This enables a screen reader to read this informa-
tion aloud for a visually impaired user.

There are several other tips like using table headers with tables of information, not just placeholders. Be careful of using the colors red and green together because of red/green colorblindness. Use sounds with care. Not everyone can hear, nor does everyone have speakers set up on their computers.

So how do you know if your web site is accessible? Go to http://webxact.watchfire.com and enter the URL for a web page. This free application will test your page and let you know where it does not comply

2. Security

Security is crucial to the success of your web site. There are several steps you can take to minimize the risk that your web sites will be subjected to a breech in security.

Security Updates
Be sure you are running the most current version of your web Server. Monitor your vendor updates, and per-form regular maintenance.

Validate User Input on the Client and the Server
Validating user input on the client is great for user experience. However, you need to validate input on the server side as well. Consider that there are tools that look and feel to your sever like a web site when in fact they are actually designed to fake input such as passwords.

Audit Logs
Maintain and review server logs to check for suspicious activity.

Common Settings
Be sure to minimize the risk to your server by minimizing the things users can do on your server. For example, don’t permit users to browse the directory structure of your site unless it’s necessary.

Lockdown Your Server
Most servers have standard development mode and then a production mode. For example, Microsoft’s Internet Information Server (IIS) has a lockdown utility that minimizes the attack surface for your web site.

3.Web Server Statistics

How many visitors do you have? What pages do they frequent? What times do they log on? Utilize a web tool to assist you in not only collecting these statistics but also analyzing and correlating them. Web tools, such as Web Trends, will aid you in collecting and utilizing this knowledge to answer these questions about your site. Build a web page that not only follows appropriate standards, but also drives repeat visitations.

4. Dynamic Technologies Styles
Are the pages within your site beginning to feel and act a little plain? Would you like more ways to format your documents and give users more interaction with them? If so, then your site could make use of languages such as Cascading Style Sheets and JavaScript.

Cascading Style Sheets (CSS) technology gives you more control of page layout and the ability to control the design of multiple pages on your site from a single file. Additionally, CSS allows you to develop more sophisticat- ed layouts, more font schemes, and even more interactivity for your pages than was possible using just HTML.

JavaScript techniques are needed to develop cutting-edge, interactive web sites. From opening windows to image-flipping and form validation, Javascript can help you build exciting, dynamic web pages.

Integrating HTML, JavaScript, and Cascading Style Sheets techniques are collectively known as Dynamic HTML or DOM Scripting. Utilizing all three languages allow you to fully exploit the capabilities of Netscape Communicator, Firefox, and Microsoft Internet Explorer.

5. Efficient Use of Appropriate Design Software

In the past, many web developers eschewed graphical web editing packages and boasted of developing pages using a simple text editor (for example, Notepad). There is still a place for text editors, but efficient designers and developers both use appropriate design software, often manually tweaking the code. Such packages offer a what- you-see-is-what-you-get (WYSIWYG) environment for designers and code-writing tools for developers. These have
the software-complete repetitive steps, and let designers and developers focus on what they do best.

There are many options, but here are some of the major ones:
Dreamweaver (Adobe, formerly Macromedia)
• The most popular package
• Offers both design (layout) and development (programming) support
• Supports all major server-side scripting languages (ColdFusion, ASP, PHP, JSP), Javascript, XML, and

ASP.NET (VB.NET and C#)
• Integrates well with Adobe Flash and Adobe Fireworks (both formerly Macromedia)

GoLive (Adobe)
• Offers both design (layout) and development (programming) support
• Supports several scripting and markup languages (PHP, JavaScript, SVG-t, SMIL)
• Integrates well with long-standing Adobe products (Photoshop, Illustrator, InDesign)

FrontPage (Microsoft)
• Offers both design (layout) and development (programming) support
• Supports Microsoft JScript and ASP.NET (VB.NET and C#)
• Integrates well with Visual Studio .NET and the Microsoft Office Suite

Microsoft Visual Studio (Microsoft)
• Primarily provides development (programming) support
• Integrated Design Environment (IDE) for developing in .NET environment
• Supports .NET languages (primarily VB.NET and C#, but other extensions for other
languages provided by third parties)

Eclipse (open source)
• Primarily provides development (programming) support
• Integrated Design Environment (IDE) for developing in any environment, but mostly
ommonly used for J2EE
• Supports a multitude of programming languages (not language-specific).

6. Standards and Browser Independence

Web site development has come a long way. There are lots of new tools that will help with web-page design, not to mention web sites that offer suggestions and ideas for making your web site absolutely incredible. Probably the biggest movement is the increasing use of Cascading Style sheets to separate page content from formatting. We are also seeing more sophisticated use of JavaScript to make pages more dynamic and, therefore, more interesting. Along with this, the World Wide Web Consortium (W3C) has instituted an effort to standardize how browsers handle the display of web pages through the use of XHTML. It is hoped that among all of these initiatives, programming for the web will become less a matter of making sure our pages work on all browsers by testing against each one, and more one of creating web pages that are useful, accessible, and exciting.

There are a vast number of resources on the web to help improve web sites. For information about the new XHTML standards, as well as help with Cascading Style Sheets and DOM Scripting (using JavaScript to make your web pages more dynamic), the W3C pages (http://www.w3.org) are invaluable. They include examples and tutorials, both of which are very well done. For some wonderful examples of Cascading Style Sheets, we recommend CSSZenGarden.com (http://www.csszengarden.com). The organization that runs this site supplies an html page with the required content. Designers are invited to create an external style sheet to format the page. New contributions are regularly posted.

Another excellent resource is http://www.dynamicdrive.com. The Internet group on Google groups, formerly Deja, http://groups.google.com/groups is helpful as well. The good news is that you can post a question through Deja/Google groups. Within 24 hours you will usually get one-to-three technically correct answers. Other potential resources include:

http://www.developer.com
This is a solid resource for most scripting/programming languages and is top-notch for Java.
http://www.codehound.com
This is another language resource and is especially helpful with Microsoft .NET technologies.
http://www.4GuysFromRolla.com
This is the definitive place to get ASP or ASP.NET information.
http://www.php.net A good resource for php.
http://www.news.com CNET news.
This site keeps you up-to-date on IT news.
http://www.theinquirer.net
This site provides a hardware outlook for six months to a year-and-a half.

7. Database Access with Server-side Scripting Languages

Static web pages are good place to start, but they quickly can become time intensive and not an efficient use of a designer or developer’s time. Database-driven web sites can refresh their own data, presenting up-to-the- minute data in way manual updates could never do. A dozen or so programmed pages can dynamically change so that they do the job of thousands of static pages. The benefits are clear: more timely information; fewer pages to maintain; and a freeing up of both designers and developers to enhance and further develop the functionality of a site, rather than its content.

But by itself, HTML is not up to this kind of job; that’s not what it was designed to do. A server-side program-
ming language is needed. There are several options, but here are a few of the major products available:

Active Server Pages (ASP) (Microsoft)
• Written using VBScript (server-side JavaScript also possible, but rare)
• Comes installed with Windows servers
• Can run in UNIX/Linux environment using Sun ONE
• Being somewhat overshadowed by ASP.NET

PHP Hypertext Preprocessor (PHP) (open source)
• C-like programming language
• No licensing cost (open-source)
• Can run on UNIX/Linux servers or Windows-based servers
• Close integration with MySQL database

ColdFusion (Adobe)
• Written using tag-based language which integrates well with web-editing software (can also be written using a scripting-like language)
• Easy to learn, quick to create and maintain pages
• Java-based architecture
• Can run on UNIX/Linux server or Windows-based servers

Java 2 Enterprise Edition (J2EE) (Sun)
• Written using cross-platform Java language
• Most often deployed on UNIX/Linux, but can run on any system
• Web page scripting using Java Server Pages (JSP)
• Most appropriate for enterprise-wide installations involving multiple servers, databases, and possibly
mainframe.

ASP.NET (Microsoft)
• Written using VB.NET or C# (other languages possible)
• Deployed in a Windows server environment
• Requires use of Visual Studio .NET for development
• Most appropriate for enterprise-wide installation with other Microsoft solutions

Perl (open source)
• Not really a scripting language (doesn’t co-habitate with HTML)
• Powerful, flexible language, good for dealing with patterns or manipulating data
• Uses less modern Common Gateway Interface (CGI) model
• More appropriate for communication between different applications on a server

Any of these will work with any Relational Database Management System (RDBMS). Here are some common ones:

Oracle (Oracle Corporation)
• Fully featured, flexible, scalable
• Works on UNIX/Linux or Windows servers

SQLServer (Microsoft)
• Fully featured, flexible, scalable
• Works on Windows servers
• Integrates well with .NET

MySQL (open source)
• Fully featured, flexible, scalable
• No licensing costs (open-source)
• Integrates especially well with PHP

Access (Microsoft)
• Friendly user-interface for database management
• Integrates well with MS Office suite
• Not fully featured, limited capabilities, only appropriate for small-scale implementations.

8. Using Image Editors for Fast Loading Graphics

Use Adobe Photoshop/Imageready or Fireworks to create the appropriate type of graphic file Typically GIF or
JPEG. There are other Image Editors available but Photoshop/Imageready and Fireworks are currently the most
popular and are considered the industry standard.

9. Site Planning, Design, and Management

“On time, within scope, and within budget” is the project management motto these days. This is also true for IT projects. IT managers can no longer live in the IT black hole. All IT requires fundamental project manage- ment best practices. Learn how to communicate with your staff. Clarify your project’s business goals to ensure that your project is aiding in the vision and meets your company’s vision. Learn how to follow through your project from inception to implementation. Apply your management skills to the concepts of web design. Apply a strategic focus within your organization to help save time and resources. Learn the benefits of various soft- ware packages to aid in efficiency.

All of these project management practices can help your web site design projects run more smoothly and ensure your web site does what it’s suppose to do.

10.Technological Flexibility

If your web application is Data driven, it is imperative that sharing information with different applications and/or platforms be done in the most flexible way possible. Transforming Data from one format to the next, however, can be arduous and considerably time consuming. Fortunately, storing data in an extensible format, and working with it using XSL, has become relatively easy.

Extensible Markup Langauge (XML) allows developers to store raw data in a text file make up with an HTML-like syntax. With the use of Extensible Style Sheet Transformations (XSL - T), Formatting Objects (XSL - FO), and CSS developers now are able to transform this raw data into an application specific format. Languages com- monly used to augment an XML application are listed below:

• Extensible Markup Langauge (XML): Used to store raw data files
• Document Type Definitions (DTDs): Used to validate XML documents
• eXtensible Stylesheet Language (XSL) and Cascading Style Sheets (CSS): Used to transform the display of XML document into an application specific format. XML utilizes complexities like XPath, functions, modes, and dynamic modification of stylesheets commonly featured in many scripting and programming languages

To Summarize
1. Make sure your web pages are accessible
2. Take steps to minimize security risks
3. Use web server statistics to determine how popular your site is
4. Utilize dynamic technologies styles
5. Be aware of web development software and how to use it efficiently to enhance your web site
6. Use XHTML, DOM Scripting, and CSS to make your site browser and rule independent
7. Choose an appropriate database to ensure it is more dynamic, and does the job of many static pages
8. Use image editors for fast-loading graphics
9. Learn to use planning and project management techniques to build great web sites with large teams
10. Use new technologies like XML, DTDs, and XSL to help your application communicate with other plat-forms in the most flexible manner

If you follow these simple rules, you too can create an excellent web site.

Understanding Domain Name System (DNS)

2007-10-24T03:48:00.000-07:00

Domain Name System (DNS) makes it possible to refer to Internet Protocol (IP) based systems (hosts) by human-friendly names (domain names). Name Resolution is the act of determining the IP address (or addresses) of a given host name.

Benefits of DNS

Domain names can be logical and easily remembered.
Should the IP address for a host change, the domain name can still resolve transparently to the user or application.

The structure of Domain Names

Domain names are separated by dots, with the topmost element on the right. Eg: www.yahoo.com . IP addresses have topmost element on the left.
Each element may be up to 63 characters long. The entire name may be atmost 255 characters long.
The right most element in a domain name is called the Top-Level Domain (TLD). Referring the above example (www.yahoo.com), TLD is 'com'.
If a domain name is not shortened, it is called the Fully Qualified Domain Name (FQDN). For example, briefcase.yahoo.com can be specified by a machine in the yahoo.com domain as either briefcase.yahoo.com (FQDN) or as briefcase.

Host names map to IP addresses in a many-to-many relationship. A host name may have one or more IP addresses. Conversely, an IP address may have multiple host names associated with it.

Hosts that are designed to perform email routing are known as mail exchangers. These machines should have special purpose records in DNS called Mail eXchanger (MX) records. When a SMTP server or mail server, needs to send mail to a remote domain, it does a DNS lookup for the Mail Exchanger (MX) of that remote domain. A domain can and should have multiple mail exchangers. Mail that cannot be sent to one mail exchanger, can instead be delivered to an alternative server, thus providing failsafe redundancy.

Different types of Domain Name Servers

Root Name server - Each top level domain (such as in,edu,com etc) has one or more root name servers which are responsible for determining where the individual records are held. These servers are fairly static and every machine on the internet has the capability of reaching any of them. A root name server is allocated like just one to three per country. For example, India has only 2 root name servers.
Authoritative Name Servers - These are the servers that the Root name servers sent queries to. These servers hold the actual information on an individual domain. This information is stored in a file called a zone file. Zone files are updated versions of the original HOSTS.TXT file.
Resolving Name Server - These are the servers that do most of the work when you are trying to get to a machine with a certain host name. Besides being responsible for looking up data, they also temporarily store the data for host names that they have searched out in a cache, which allows them to speed up the resolution for host names that are frequently visited.

Zone

A zone keeps the information about the domain database. It does this by maintaining two types of files:

Zone File - It is used to map host names to address, to identify the mail servers, and to provide other domain information.

Reverse Zone File - This file is responsible for mapping IP address to host names, which is exactly the opposite of what the zone file does.

Note: The zone file and the reverse zone file has to be maintained by the user.

Name Server Hierarchy

Master Name Server - Also called primary server. This contains the master copy of data for a zone.

Slave Name Server - Also known as secondary server. This provides a backup to the master name server. All slave servers maintain synchronization with their master name server.

A zone may have multiple slave servers. But there will be only one master name server per zone.

Apache : Name-based Vs IP Based Virtual Hosting

Often when, you attend interviews for network administration related jobs , the one question you may encounter while discussing about web servers is the difference between name-based and IP based virtual hosting. Here I will explain the difference between the two.

In IP-based virtual hosting, you are running more than one web site on the same server machine, but each web site has its own IP address. In order to do this, you have to first tell your operating system about the multiple IP addresses. See here configuring multiple IP addresses on a single NIC . You also need to put each IP in your DNS, so that it will resolve to the names that you want to give those addresses .

In Name-based virtual hosting, you host multiple websites on the same IP address. But for this to succeed, you have to put more than one DNS record for your IP address in the DNS database. This is done using CNAME tag in BIND. You can have as many CNAME(s) as you like pointing to a particular machine. Of course, you also have to uncomment the NameVirtualHost section in httpd.conf file and point it to the IP address of your machine.


#FILE: httpd.conf
...
NameVirtualHost 192.168.0.1
...

Setting up multiple IP addresses on a single NIC

In linux, you can bind multiple IP addresses on a single NIC. This is usually done in case you are using your linux machine as a webserver and is hosting multiple domains and you want to bind each domain to a unique IP address. This is how it is done.

Let us assume that you already have a NIC which is bound with a static IP address. Then you will have a file called /etc/sysconfig/network-scripts/ifcfg-eth0 .My ifcfg-eth0 file has the following entries:

# File: ifcfg-eth0
DEVICE=eth0
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.0.1
NETMASK=255.255.255.0
BROADCAST=192.168.0.255
NETWORK=192.168.0.0
HWADDR=00:80:48:34:C2:84

Now to bind another IP address to the same NIC, I create a copy of the above file ifcfg-eth0 and name it as ifcfg-eth0:1

# cd /etc/sysconfig/networking-scripts
# cp ifcfg-eth0 ifcfg-eth0:1

Now just change the values of the DEVICE and IPADDR in the file as follows:

# File: ifcfg-eth0:1
DEVICE=eth0:1
ONBOOT=yes
BOOTPROTO=static
IPADDR=192.168.0.5
NETMASK=255.255.255.0
BROADCAST=192.168.0.255
NETWORK=192.168.0.0
HWADDR=00:80:48:34:C2:84

And lastly, restart the networking service. If you are using RedHat, then it is as simple as :
# service network restart

How to install a Network card in linux

There are different ways of installing a network card in linux - and that too depending on the linux distribution that you are using. I will explain each one of these methods here.

1) The Manual method

First open the computer case and insert the network card into an empty PCI slot. Then boot up your machine to load linux. In linux login as root and then navigate to the directory /lib/modules/kernel_version_number/net/ . Here you will find the modules supported by your system. Assuming that you have a 3Com ethernet card, in which case, the module name is 3c59x , you have to add this in the /etc/modules.conf file to let the machine detect the card each time the machine boots.

#File: /etc/modules.conf
alias eth0 3c59x

Note: If you have only one network card, it is known by the name eth0, the succeeding network cards in your computer go by the name eth1, eth2 ... and so on.

Now you have to load the module into the kernel.

root# /sbin/insmod -v 3c59x

Next configure an IP address for the network card using ifconfig or netconfig or any other method if your machine gets its IP address from a DHCP server. Eg:

root# ifconfig eth0 192.168.1.5 netmask 255.255.255.0 broadcast 192.168.1.255

2) The Easy way
RedHat/Fedora distributions of linux ships with Kudzu a device detection program which runs during systems initialization (/etc/rc.d/init.d/kudzu). This can detect a newly installed NIC and load the appropriate driver. Then use the program /usr/sbin/netconfig to configure the IP address and network settings. The configuration will be stored so that it will be utilized upon system boot.

How to Assign an IP address

Computers may be assigned a static IP address or assigned one dynamically (via DHCP). Here I will explain the steps needed to assign an IP address to your NIC.

Choose one of the following methods:

=> Dynamic Host Configuration Protocol (DHCP) is a protocol used by networked computers (clients) to obtain IP addresses and other parameters such as the default gateway, subnet mask, and IP addresses of DNS servers from a DHCP server.
Command line :

/sbin/ifconfig eth0 192.168.1.3 netmask 255.255.255.0 broadcast 192.168.1.255

GUI tool : You can use the GUI tool /usr/bin/neat - Gnome GUI network administration tool. It handles all interfaces and configures for both static assignment as well as dynamic assignment using DHCP.

Console tool : /usr/sbin/netconfig (Only seems to work for the first network interface eth0 but not eth1,...)

The ifconfig command does NOT store this information permanently. Upon reboot this information is lost. (Manually add the commands to the end of the file /etc/rc.d/rc.local to execute them upon boot.) The command netconfig and /usr/bin/neat make permanent changes to system network configuration files located in /etc/sysconfig/network-scripts/ , so that this information is retained.

The Red Hat configuration tools store the configuration information in the file /etc/sysconfig/network. They will also allow one to configure routing information.

# File: /etc/sysconfig/network
# Static IP address Configuration:
NETWORKING=yes
HOSTNAME=my-hostname # Hostname is defined here and by command hostname
FORWARD_IPV4=true # True for NAT firewall gateways and linux routers. False for
# everyone else - desktops and servers.
GATEWAY="XXX.XXX.XXX.YYY" # Used if your network is connected to another
# network or the internet.

# Gateway not defined here for DHCP.

# Or for DHCP configuration: in the same file /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=my-hostname # Hostname is defined here and by command hostname
# Gateway is assigned by DHCP.

# File: /etc/sysconfig/network-scripts/ifcfg-eth0
# Static IP address configuration:
DEVICE=eth0
BOOTPROTO=static
BROADCAST=XXX.XXX.XXX.255
IPADDR=XXX.XXX.XXX.XXX
NETMASK=255.255.255.0
NETWORK=XXX.XXX.XXX.0
ONBOOT=yes

# OR for DHCP configuration:
DEVICE=eth0
ONBOOT=yes
BOOTPROTO=dhcp

Used by script /etc/sysconfig/network-scripts/ifup to bring the various network interfaces on-line.

To disable DHCP change BOOTPROTO=dhcp to BOOTPROTO=none
In order for updated information in any of these files to take effect, one must issue the command:
root# service network restart

Open Source Law

2007-10-24T03:47:00.000-07:00

Columbia University Law Professor Eben Moglen today announced the formation of the Software Freedom Law Center, whose mission is to provide pro-bono legal services globally to eligible non-profit open source software projects and developers.

"As the popularity and use of free and open source software increases and proprietary software development models are threatened, providing necessary legal services to open source developers is becoming increasingly important to prevent liability and other legal issues from interfering with its success," Moglen said. "The Law Center is being established to provide legal services to protect the legitimate rights and interests of free and open source software projects and developers, who often do not have the means to secure the legal services they need."

OSDL has raised more than $4 million for a newly-established IP fund that will provide the seed money for the new and independent legal center based in New York. Last year, OSDL announced a separate $10 million Linux Legal Defense Fund to provide legal support for Linus Torvalds and end user companies subjected to Linux-related litigation by the SCO Group. The new Law Center announced today will be an independent organization not affiliated with OSDL.

"OSDL is committed to supporting initiatives such as the Law Center to help protect the legitimate development and use of Linux and open source software," said Stuart Cohen, CEO of OSDL. "We encourage other companies and organizations like OSDL who are dedicated to securing the future of open source software to contribute to the Law Center and participate in its good works."

Overseeing the Law Center will be a distinguished board of directors comprised of Moglen; Diane Peters, General Counsel at OSDL; Daniel Weitzner, Principal Research Scientist at MIT's Computer Science and Artificial Intelligence Laboratory; and Lawrence Lessig, Stanford Law Professor and author.

"Both free and open source software face many emerging legal threats," said Lessig. "We should be skeptical of legal mechanisms that enable those most threatened by the success of open source and free software to resist its advance. The Law Center will serve as important support for the free and open source communities and for those that benefit from free and open source software."

Moglen, regarded as one of the world's leading experts on copyright law as applied to software, will run the new Law Center from its headquarters in New York City. The Law Center will initially have two full-time intellectual property attorneys on staff and expects to expand to four attorneys later this year. Initial clients for the Law Center include the Free Software Foundation and the Samba Project.

"Free software projects often face legal issues that need expert advice, but it can sometimes be difficult or prohibitively costly to obtain that advice through traditional legal channels," said Andrew Tridgell, head of the Samba project. "We are delighted that the Free Software Law Center is being setup under Eben Mogeln's excellent guidance. I think this is an important milestone in the maturity of the free software community."

Legal services provided to eligible individuals and projects include asset stewardship, licensing, license defense and litigation support, and legal consulting and lawyer training. The Law Center will be software license neutral and intends to participate directly in work currently underway around revisions to the GNU General Public License (GPL) with the Free Software Foundation. The Law Center will also work on issues around the proliferation of open source licenses.

The Law Centeris dedicated to assisting non-profit open source developers and projects who do not otherwise have access to necessary legal services.

OSDL is dedicated to accelerating the growth and adoption of Linux in the enterprise.

Ajax Tutorials for Beginners

2007-10-24T03:46:00.002-07:00

This is the first in a three-part series of the top Ajax tutorials from around the web. Part one is for beginners, how to build your first ajax application, and understanding how it works. Part two will be for novices who have experience with ajax, but would like to take their skills to the next level. And part three will be for experts who want to... build their own gmail application? Why not. So we'll get started from the very beginning.

What is Ajax? Ajax stands for Asynchronous JavaScript and XML. In a nutshell, it is the use of the nonstandard XMLHttpRequest() object to communicate with server-side scripts. It can send as well as receive information in a variety of formats, including XML, HTML, and even text files. Ajax’s most appealing characteristic, however, is its “asynchronous” nature, which means it can do all of this without having to refresh the page. This allows you to update portions of a page based upon user events. You can: Make requests to the server without reloading the page and Parse and work with XML documents.

Our first tutorial is from the Mozilla development page. This is the number one example because it provides you will all the information you need to make a basic "hello world" ajax app. By the end of this article you will be able to make simple ajax application on your own.

Our next tutorial is from W3schools. Here we take what you learned in the previous article and we incorporate it into forms. So you can put together a log-in or registration form, and check for valid credentials without navigating away from the current page. Very convenient. The overall trick is not to put your input variables into a form tag. and your submit button should contain an onSubmit="javascript:function()"> so instead of directing the user to a new page, the javascript call will check the credentials in real time.

Now let's migrate into XML a little bit. Web Reference has a great tutorial on the basics of using XML with Ajax. The method they use for parsing XML files is very similar to the method's used in the above two examples, so make sure you totally understand how XMLHttpRequest works.

function ajaxRead(file){
var xmlObj = null;
if(window.XMLHttpRequest){
   xmlObj = new XMLHttpRequest();
} else if(window.ActiveXObject){
   xmlObj = new ActiveXObject("Microsoft.XMLHTTP");
} else {
   return;
}
xmlObj.onreadystatechange = function(){
 if(xmlObj.readyState == 4){
    updateObj('xmlObj', xmlObj.responseXML.getElementsByTagName('data')[0].firstChild.data);
  }
 }
 xmlObj.open ('GET', file, true);
 xmlObj.send ('');
}
function updateObj(obj, data){
document.getElementById(obj).firstChild.data = data;
}

Although javascript is required to complete the Ajax request, you don't have to know a lot of javascript to make good use of Ajax. For example you can make a form, and onSubmit use javascript (document.getElementById()) to get the content of the input id's, and send them to a php file. Let PHP parse all the data, and return the result to javascript to display. Very minimal javascript is required, but if you know javascript is can be extremely helpful when you get into novice and advanced tutorials. Developer.com has a great article called "Ajax from Scratch" that shows different methods than the above ones, for making ajax requests. They also have a lot of more advanced javascript helping them do this, so if you are a fan of (or at least understand) javascript, check it out.

1 function Mutex( cmdObject, methodName ) {
2   // define static variable and method
3   if (!Mutex.Wait) Mutex.Wait = new Map();
4   Mutex.SLICE = function( cmdID, startID ) {
5     Mutex.Wait.get(cmdID).attempt( Mutex.Wait.get(startID) );
6   }
7   // define instance method
8   this.attempt = function( start ) {
9     for (var j=start; j; j=Mutex.Wait.next(j.c.id)) {
10       if (j.enter || (j.number (j.number < number ="=" number =" 0;" c        =" cmdObject;" methodid =" methodName;" enter    =" true;" number   =" (new" enter    =" false;">

Now the question is, what are you going to use for the back end of the ajax requests? The two most popular methods are ASP.NET and PHP. Personally I prefer PHP whether I'm using ajax or not, but that doesn't mean asp.net doesn't have it's advantages. Ajax Projects has a nice "hello world" example using ASP.NET with your new ajax requests. Before moving on to the PHP back end of things, you should also check out "Ajax and PHP without XmlHttpRequest object" from PHPit, it outlines a nice alternative to using the traditional ajax method.

url = document.location.href;
xend = url.lastIndexOf("/") + 1;
var base_url = url.substring(0, xend);

function ajax_do (url) {
     if (url.substring(0, 4) != 'http') {
             url = base_url + url;
     }
     var jsel = document.createElement('SCRIPT');
     jsel.type = 'text/javascript';
     jsel.src = url;
     document.body.appendChild (jsel);
}

Now on to some "good stuff". Lets put what you just learned to work with something useful. First we'll look at creating a comment system for your website. Comments are a great way for users to interact with a website, and often times it's easier to let them add a comment without leaving the page they're on. To do this we'll take a look at the Ajax Feedback Mechanism from iBegin. Although it's pre-written code you can still use and learn from it. There are a lot of advanced effects in this library (fading in and out when adding / removing comments) you will understand the basic concept of taking data from a from, sending it to a backend PHP script, and showing the user the new information (e.g. "Successful log in" or "Failed log in") without leaving the web page. Web Pasties has a similar walk-thru (except no download is required) that goes through form creation and how to store, check and display data from the current page. Both of these articles are great resources to get you going on your first ajax project.

Ajax isn't limited to just input boxes either. For example, phpRiot has a walk-thru that shows you out to create real-time sortable lists. You can click and drag items into a new order (much like you can in Netvibes or Google IG). This is very nifty for sites with lots of content in one area. This way users can decide what they want to see first. The Tool Man takes the last example a few steps further by adding the ability to add multiple lists to one sorting method, add multi-row and column support and more.

That wraps it up for part one of this three-part series. From this article you should know how to create an XMLHttpRequest object, how to grab data from inside text boxes (and other input areas), and how to display that information for the user in real-time without leaving your web page. In part two we will pick up where we left off here, and go into more advanced options, so be sure you know all the information here before reading on!

Mistakes made when developing with Ajax

2007-10-24T03:46:00.001-07:00

Using Ajax for the sake of Ajax.

Sure Ajax is cool, and developers love to play with cool technology, but Ajax is a tool, not a toy. A lot of Ajax isn’t seriously needed to improve usability but rather experiments in what Ajax can do or trying to fit Ajax somewhere where it isn’t needed.

Breaking the back button

The back button is a great feature of the standard web site user interface. Unfortunately, the back button doesn’t mesh very well with Javascript. Keeping back button functionality is one reason not to go with a pure Javascript web app.

Keep in mind however that good web design provides the user with everything they would need to successfully navigate your site, and never relies on web browser controls.

Not giving immediate visual cues for clicking widgets

If something I’m clicking on is triggering Ajax actions, you have to give me a visual cue that something is going on. An example of this is GMail loading button that is in the top right. Whenever I do something in GMail, a little red box in the top right indicates that the page is loading, to make up for the fact that Ajax doesn’t trigger the normal web UI for new page loading.

Leaving offline people behind

As web applications push the boundaries further and further, it becomes more and more compelling to move all applications to the web. The provisioning is better, the world-wide access model is great, the maintenance and configuration is really cool, the user interface learning curve is short.

However, with this new breed of Ajax applications, people who have spotty internet connections or people who just don’t want to switch to the web need to be accomodated as well. Just because technology ‘advances’ doesn’t mean that people are ready and willing to go with it. Web application design should at least consider offline access. With GMail it’s POP, Backpackit has SMS integration. In the Enterprise, it’s web-services.
Don’t make me wait

With Firefox tabs, I can manage various waits at websites, and typically I only have to wait for a page navigation. With AJAX apps combined with poor network connectivity/bandwidth/latency I can have a really terrible time managing an interface, because every time I do something I have to wait for the server to return a response. However, remember that the ‘A’ in AJAX stands for ‘Asynchronous’, and the interaction can be designed so that the user is not prevented from continuing to work on the page while the earlier request is processed.

Sending sensitive information in the clear

The security of AJAX applications is subject to the same rules as any web application, except that once you can talk asynchronously to the server, you may tend to write code that is very chatty in a potentially insecure way. All traffic must be vetted to make sure security is not compromised.

Assuming AJAX development is single platform development.

Ajax development is multi-platform development. Ajax code will run on IE’s javascript engine, Spidermonkey (Mozilla’s js engine), Rhino (a Java js implementation, also from Mozilla), or other minor engines that may grow into major engines. So it’s not enough just to code to JavaScript standards, there needs to be real-world thorough testing as well. A major obstacle in any serious Javascript development is IE’s buggy JS implementation, although there are tools to help with IE JS development.

Forgetting that multiple people might be using the same application at the same time

In the case of developing an Intranet type web application, you have to remember that you might have more than one person using the application at once. If the data that is being displayed is dynamically stored in a database, make sure it doesn’t go “stale” on you.

Too much code makes the browser slow

Ajax introduces a way to make much more interesting javascript applications, unfortunately interesting often means more code running. More code running means more work for the browser, which means that for some javascript intensive websites, especially inefficiently coded ones, you need to have a powerful CPU to keep the functionality zippy. The CPU problem has actually been a limit on javascript functionality in the past, and just because computers have gotten faster doesn’t mean the problem has disappeared.

Not having a plan for those who do not enable or have JavaScript.

According to the W3 schools browser usage statistics, which if anything are skewed towards advanced browsers, 11% of all visitors don’t have JavaScript. So if your web application is wholly dependent on JavaScript, it would seem that you have potentially cut a tenth of your audience.

Blinking and changing parts of the page unexpectedly

The first A in Ajax stands for asynchronous. The problem with asynchronous messages is that they can be quite confusing when they pop in unexpectedly. Asynchronous page changes should only ever occur in narrowly defined places and should be used judiciously, flashing and blinking in messages in areas I don’t want to concentrate on harkens back to days of the html blink tag. “Yellow Fade”, “One Second Spotlight” and other similar techniques are used to indicate page changes unobtrusively.

Not using links I can pass to friends or bookmark

Another great feature of websites is that I can pass URLs to other people and they can see the same thing that I’m seeing. I can also bookmark an index into my site navigation and come back to it later. Javascript, and thus Ajax applications, can cause huge problems for this model of use. Since the Javascript is dynamically generating the page instead of the server, the URL is cut out of the loop and can no longer be used as an index into navigation. This is a very unfortunate feature to lose, many Ajax webapps thoughtfully include specially constructed permalinks for this exact reason.

Blocking Spidering

Ajax applications that load large amounts of text without a reload can cause a big problem for search engines. This goes back to the URL problem. If users can come in through search engines, the text of the application needs to be somewhat static so that the spiders can read it.

Asynchronously performing batch operations

Sure with Ajax you can make edits to a lot of form fields happen immediately, but that can cause a lot of problems. For example if I check off a lot of check boxes that are each sent asynchronously to the server, I lose my ability to keep track of the overall state of checkbox changes and the flood of checkbox change indications will be annoying and disconcerting.

Scrolling the page and making me lose my place

Another problem with popping text into a running page is that it can effect the page scroll. I may be happily reading an article or paging through a long list, and an asynchronous javascript request will decide to cut out a paragraph way above where I’m reading, cutting my reading flow off. This is obviously annoying and it wastes my time trying to figure out my place. But then again, that would be a very stupid way to program a page, with or without AJAX.

Inventing new UI conventions

A major mistake that is easy to make with Ajax is: ‘click on this non obvious thing to drive this other non obvious result’. Sure, users who use an application for a while may learn that if you click and hold down the mouse on this div that you can then drag it and permanently move it to this other place, but since that’s it’s not in the common user experience, you increase the time and difficulty of learning the application, which is a major negative for any application. On the plus side, intuitiveness is a function of learning, and AJAX is popularising many new conventions which will become intuitive as time goes by. The net result will be greater productivity once the industry gets over the intuitiveness hump.

Character Sets

One big problem with using AJAX is the lack of support for character sets. You should always set the content character set on the server-side as well as encoding any data sent by Javascript. Use ISO-8859-1 if you use plain english, or UTF-8 if you use special characters, like æ, ø and å (danish special characters) Note: it is usually a good idea to go with utf-8 nowadays as it supports many languages).

Changing state with links (GET requests)

The majority of Ajax applications tend to just use the GET method when working with AJAX. However, the W3C standards state that GET should only be used for retrieving data, and POST should only be used for setting data. Although there might be no noticable difference to the end user, these standards should still be followed to avoid problems with robots or programs such as Google Web Accelerator.

Not cascading local changes to other parts of the page

Since Ajax/Javascript gives you such specific control over page content, it’s easy to get too focused on a single area of content and miss the overall integrated picture. An example of this is the Backpackit title. If you change a Backpackit page title, they immediately replace the title, they even remember to replace the title on the right, but they don’t replace the head title tag with the new page title. With Ajax you have to think about the whole picture even with localized changes.

Problem reporting

In a traditional server-side application, you have visibility into every exception, you can log all interesting events and benchmarks, and you can even record and view (if you wish) the actual HTML that the browser is rendering. With client-side applications, you may have no idea that something has gone wrong if you don’t know how to code correctly and log exceptions from the remotely called pages to your database.

Return on Investment

Sometimes AJAX can impressively improve the usability of an application (a great example is the star-rating feedback on Netflix), but more often you see examples of expensive rich-client applications that were no better than the plain HTML versions.

Mimicing browser page navigation behavior imperfectly

One example of this is blinklist Ajax paging mechanism on the front page. As you click to see another page of links, ajax fills in the next page. Except that if you are used to a browser experience, you probably expect to go to the top of the page when you hit next page, something JavaScript driven page navigation doesn’t do. BlinkList actually anticipates this and tries to counteract by manipulating your scrolling to scroll upwards until you hit the top. Except this can be slow and if you try scrolling down you will fight the upwards scrolling JavaScript and it won’t let you scroll down. But then again, that is very stupid way to program a page, with or without AJAX.

Another Tool

It seems everyone has forgotten that Ajax is just another tool in the toolbox for Web Design. You can use it or not and misuse it or not. The old 80/20 rule always applies to applications (if you cover 80% of what all users want/need then you have a viable app) and if you lose 11% of your audience because they don’t switch on their javascript then you have to ask yourself if changing your app is worth capturing that 11% or stick with 89% that are currently using it and move on to something else. Also web apps should take advantage of all tricks to enable them to function quickly and efficiently. If that means using javascript for some part, Ajax for another and ASP callbacks for a third, so be it.

IE7 and AJAX. IE7 has added native XMLHTTPRequest object

2007-10-24T03:45:00.000-07:00

Native XMLHTTPRequest object

I’m excited to mention that IE7 will support a scriptable native version of XMLHTTP. This can be instantiated using the same syntax across different browsers and decouples AJAX functionality from an ActiveX enabled environment.

What is XMLHTTP?

XMLHTTP was first introduced to the world as an ActiveX control in Internet Explorer 5.0. Over time, this object has been implemented by other browsing platforms, and is the cornerstone of “AJAX” web applications. The object allows web pages to send and receive XML (or other data) via the HTTP protocol. XMLHTTP makes it possible to create responsive web applications that do not require redownloading the entire page to display new data. Popular examples of AJAX applications include the Beta version of Windows Live Local, Microsoft Outlook Web Access, and Google’s GMail.

Charting the changes: XMLHTTP in IE7 vs. IE6

In IE6 and below, XMLHTTP is implemented as an ActiveX object provided by MSXML.

In IE7, XMLHTTP is now also exposed as a native script object. Users and organizations that choose to disable ActiveX controls can still use XMLHTTP based web applications. (Note that an organization may use Group Policy or IE Options to disable the new native XMLHTTP object if desired.) As part of our continuing security improvements we now allow clients to configure and customize a security policy of their choice and simultaneously retain functionality across key AJAX scenarios.

IE7’s implementation of the XMLHTTP object is consistent with that of other browsers, simplifying the task of cross-browser compatibility. Using just a bit of script, it’s easy to build a function which works with any browser that supports XMLHTTP:

if (window.XMLHttpRequest){

// If IE7, Mozilla, Safari, etc: Use native object
var xmlHttp = new XMLHttpRequest()

}
else
{
if (window.ActiveXObject){

// ...otherwise, use the ActiveX control for IE5.x and IE6
var xmlHttp = new ActiveXObject("Microsoft.XMLHTTP");
}

}

Note that IE7 will still support the legacy ActiveX implementation of XMLHTTP alongside the new native object, so pages currently using the ActiveX control will not require rewrites.

I look forward to hearing any feedback and suggestions.

How to reduce page loading time.

2007-10-24T03:44:00.000-07:00

It is widely accepted that fast-loading pages improve the user experience. In recent years, many sites have started using AJAX techniques to reduce latency. Rather than round-trip through the server retrieving a completely new page with every click, often the browser can either alter the layout of the page instantly or fetch a small amount of HTML, XML, or javascript from the server and alter the existing page. In either case, this significantly decreases the amount of time between a user click and the browser finishing rendering the new content.

However, for many sites that reference dozens of external objects, the majority of the page load time is spent in separate HTTP requests for images, javascript, and stylesheets. AJAX probably could help, but speeding up or eliminating these separate HTTP requests might help more, yet there isn't a common body of knowledge about how to do so.

While working on optimizing page load times for a high-profile AJAX application, I had a chance to investigate how much I could reduce latency due to external objects. Specifically, I looked into how the HTTP client implementation in common browsers and characteristics of common Internet connections affect page load time for pages with many small objects.

Try some of the following:

Turn on HTTP keepalives for external objects. Otherwise you add an extra round-trip for every HTTP request. If you are worried about hitting global server connection limits, set the keepalive timeout to something short, like 5-10 seconds. Also look into serving your static content from a different webserver than your dynamic content. Having thousands of connections open to a stripped down static file webserver can happen in like 10 megs of RAM total, whereas your main webserver might easily eat 10 megs of RAM per connection.
Load fewer external objects. Due to lower request overhead, one bigger file just loads faster than two smaller ones half its size. Figure out how to globally reference the same one or two javascript files and one or two external stylesheets instead of many; if you have more, try preprocessing them when you publish them. If your UI uses dozens of tiny GIFs all over the place, consider switching to a much cleaner CSS-based design which probably won't need so many images. Or load all of your common UI images in one request using a technique called "CSS sprites".
If your users regularly load a dozen or more uncached or uncacheable objects per page, consider evenly spreading those objects over four hostnames. This usually means your users can have 4x as many outstanding connections to you. Without HTTP pipelining, this results in their average request latency dropping to about 1/4 of what it was before.

When you generate a page, evenly spreading your images over four hostnames is most easily done with a hash function, like MD5. Rather than having all tags load objects from http://static.example.com/, create four hostnames (e.g. static0.example.com, static1.example.com, static2.example.com, static3.example.com) and use two bits from an MD5 of the image path to choose which of the four hosts you reference in the tag. Make sure all pages consistently reference the same hostname for the same image URL, or you'll end up defeating caching.

Beware that each additional hostname adds the overhead of an extra DNS lookup and an extra TCP three-way handshake. If your users have pipelining enabled or a given page loads fewer than around a dozen objects, they will see no benefit from the increased concurrency and the site may actually load more slowly. The benefits only become apparent on pages with larger numbers of objects. Be sure to measure the difference seen by your users if you implement this.
Possibly the best thing you can do to speed up pages for repeat visitors is to allow static images, stylesheets, and javascript to be unconditionally cached by the browser. This won't help the first page load for a new user, but can substantially speed up subsequent ones.
Set an Expires header on everything you can, with a date days or even months into the future. This tells the browser it is okay to not revalidate on every request, which can add latency of at least one round-trip per object per page load for no reason.

Instead of relying on the browser to revalidate its cache, if you change an object, change its URL. One simple way to do this for static objects if you have staged pushes is to have the push process create a new directory named by the build number, and teach your site to always reference objects out of the current build's base URL. (Instead of you'd use . When you do another build next week, all references change to .) This also nicely solves problems with browsers sometimes caching things longer than they should -- since the URL changed, they think it is a completely different object.

If you conditionally gzip HTML, javascript, or CSS, you probably want to add a "Cache-Control: private" if you set an Expires header. This will prevent problems with caching by proxies that won't understand that your gzipped content can't be served to everyone. (The Vary header was designed to do this more elegantly, but you can't use it because of IE brokenness.)

For anything where you always serve the exact same content when given the same URL (e.g. static images), add "Cache-Control: public" to give proxies explicit permission to cache the result and serve it to different users. If a local cache has the content, it is likely to have much less latency than you; why not let it serve your static objects if it can?

Avoid the use of query params in image URLs, etc. At least the Squid cache refuses to cache any URL containing a question mark by default. I've heard rumors that other things won't cache those URLs at all, but I don't have more information.
On pages where your users are often sent the exact same content over and over, such as your home page or RSS feeds, implementing conditional GETs can substantially improve response time and save server load and bandwidth in cases where the page hasn't changed.

When serving a static files (including HTML) off of disk, most webservers will generate Last-Modified and/or ETag reply headers for you and make use of the corresponding If-Modified-Since and/or If-None-Match mechanisms on requests. But as soon as you add server-side includes, dynamic templating, or have code generating your content as it is served, you are usually on your own to implement these.

The idea is pretty simple: When you generate a page, you give the browser a little extra information about exactly what was on the page you sent. When the browser asks for the same page again, it gives you this information back. If it matches what you were going to send, you know that the browser already has a copy and send a much smaller 304 (Not Modified) reply instead of the contents of the page again. And if you are clever about what information you include in an ETag, you can usually skip the most expensive database queries that would've gone into generating the page.
Minimize HTTP request size. Often cookies are set domain-wide, which means they are also unnecessarily sent by the browser with every image request from within that domain. What might've been a 400 byte request for an image could easily turn into 1000 bytes or more once you add the cookie headers. If you have a lot of uncached or uncacheable objects per page and big, domain-wide cookies, consider using a separate domain to host static content, and be sure to never set any cookies in it.
Minimize HTTP response size by enabling gzip compression for HTML and XML for browsers that support it. For example, the 17k document you are reading takes 90ms of the full downstream bandwidth of a user on 1.5Mbit DSL. Or it will take 37ms when compressed to 6.8k. That's 53ms off of the full page load time for a simple change. If your HTML is bigger and more redundant, you'll see an even greater improvement.

If you are brave, you could also try to figure out which set of browsers will handle compressed Javascript properly. (Hint: IE4 through IE6 asks for its javascript compressed, then breaks badly if you send it that way.) Or look into Javascript obfuscators that strip out whitespace, comments, etc and usually get it down to 1/3 to 1/2 its original size.
Consider locating your small objects (or a mirror or cache of them) closer to your users in terms of network latency. For larger sites with a global reach, either use a commercial Content Delivery Network, or add a colo within 50ms of 80% of your users and use one of the many available methods for routing user requests to your colo nearest them.
Regularly use your site from a realistic net connection. Convincing the web developers on my project to use a "slow proxy" that simulates bad DSL in New Zealand (768Kbit down, 128Kbit up, 250ms RTT, 1% packet loss) rather than the gig ethernet a few milliseconds from the servers in the U.S. was a huge win. We found and fixed a number of usability and functional problems very quickly.

To implement the slow proxy, I used the netem and HTB kernel modules available in the Linux 2.6 kernel, both of which are set up with the tc command line tool. These offer the most accurate simulation I could find, but are definitely not for the faint of heart. I've not used them, but supposedly Tamper Data for Firefox, Fiddler for Windows, and Charles for OSX can all rate-limit and are probably easier to set up, but they may not simulate latency properly.
Use Google's Load Time Analyzer extension for Firefox from a realistic net connection to see a graphical timeline of what it is doing during a page load. This shows where Firefox has to wait for one HTTP request to complete before starting the next one and how page load time increases with each object loaded. The Tamper Data extension can offer similar data in less easy to interpret form. And the Safari team offers a tip on a hidden feature in their browser that offers some timing data too.

Or if you are familiar with the HTTP protocol and TCP/IP at the packet level, you can watch what is going on using tcpdump, ngrep, or ethereal. These tools are indispensible for all sorts of network debugging.
Try benchmarking common pages on your site from a local network with ab, which comes with the Apache webserver. If your server is taking longer than 5 or 10 milliseconds to generate a page, you should make sure you have a good understanding of where it is spending its time.

If your latencies are high and your webserver process (or CGI if you are using that) is eating a lot of CPU during this test, it is often a result of using a scripting language that needs to recompile your scripts with every request. Software like eAccelerator for PHP, mod_perl for perl, mod_python for python, etc can cache your scripts in a compiled state, dramatically speeding up your site. Beyond that, look at finding a profiler for your language that can tell you where you are spending your CPU. If you fix that, your pages will load faster and you'll be able to handle more traffic with fewer machines.

If your site relies on doing a lot of database work or some other time-consuming task to generate the page, consider adding server-side caching of the slow operation. Most people start with writing a cache to local memory or local disk, but that starts to fall down if you expand to more than a few web server machines. Look into using memcached, which essentially creates an extremely fast shared cache that's the combined size of the spare RAM you give it off of all of your machines. It has clients available in most common languages.
(Optional) Petition browser vendors to turn on HTTP pipelining by default on new browsers. Doing so will remove some of the need for these tricks and make much of the web feel much faster for the average user. (Firefox has this disabled supposedly because some proxies, some load balancers, and some versions of IIS choke on pipelined requests. But Opera has found sufficient workarounds to enable pipelining by default. Why can't other browsers do similarly?)

How do I find the geographical location of a host, given its IP address ?

2007-10-24T03:43:00.002-07:00

In general, it is impossible - IP addresses are allocated arbitrarily, as there's no inherent connection between an IP address and it's physical location, and there's no reliable method to do the trick.

Yet, doing some detective work could help. Try following methods :

Note the following links for reference :

A complete list of country codes

http://www.iana.org/domain-names.htm

http://www.ics.uci.edu/pub/websoft/wwwstat/country-codes.txt

A complete list of U.S. state abbreviation

http://www.usps.gov/ncsc/lookups/abbr_state.txt

A complete list of airport codes

http://www.aviationjobsonline.com/airports/citycode.html

Microsoft's TerraServer - satellites pictures of geographical areas

http://www.terraserver.microsoft.com/
Use reverse DNS to find out the host's name. This item could supply some clues that could help.

E.g. given the IP address 132.74.18.2, the command 'nslookup 132.74.18.2' translates the address to construct.haifa.ac.il gives two hints -
1. The TLD is .il, which hints the host is in Israel.
2. The next two domains are haifa.ac, so this host belongs to the 'haifa' academical institute (a university, in this case). The Haifa university happens to be in the city Haifa.
Reverse DNS translation doesnt always work - it depends on the host's [the host with the given IP address] DNS server's correct configuration.

Another trick is to execute a whois request on the IP address. Try to direct the whois query to whois.arin.net - if it doesn't have the reply it will tell you to query either whois.apnic.net or whois.ripe.net

Notice that a host in one domain might be hosted in another country. This is due to both virtual hosting, where a domain of a company from one country or region, might be hosted where hosting is cheap.

Also notice that the .org, .com, and even .edu domains does not imply the host is in the U.S., as many of those domains belong to companies that are either not U.S. based, or are international, and might have some hosts all over the world.
Some hosts support a DNS extension which allows for hosts to enter their geographical location into their DNS record, based on an extension to DNS described in RFC 1876.

For further information see - http://www.ckdhr.com/dns-loc/

Another attempt to express a host's geographical location via DNS is done in RFC 1712. Both RFCs define a DNS Resource Record to contain the geographical location.
Visit the host's web server. A web site will sometimes contain hints regarding the site's location.

E.g. for construct.haifa.ac.il, you can find info at both http://www.haifa.ac.il/ and http://www.ac.il/
Use whois. The whois database contains administrative contact info for all domains, filled in during domain registration time, and updated from time to time. This admin info could give some hints.

The whois database is not highly reliable - if an address belongs to a large & responsible company, the company will supply reliable info and update it, but as domain name registrators do not insist on keeping the database accurate and current, the data might be incorrect.

The IP to Lat/Long page will attempt to display the same information in a graphical representation.

http://cello.cs.uiuc.edu/cgi-bin/slamm/ip2ll/

The Allwhois.com page allows whois requests for many countries.

http://www.allwhois.com/

A list of whois servers, collected by Matt Power, is available at ftp://sipb.mit.edu/pub/whois/whois-servers.list

Note that the data is usually given for the owners' main branch or contact points, but the IP addresses might be allocated to hosts that may be located at a different location(s).

Use traceroute. The names of the routers through which packets flow from your [or any] host to the host with the given IP address might hint at the geographical path which the packets follow, and at the final destination's physical location.


E.g. > traceroute www.mit.edu
traceroute to DANDELION-PATCH.MIT.EDU(18.181.0.31), ...
1 teg.technion.ac.il (132.68.7.254)              2 ms   1 ms   1 ms
2 tau-smds.man.ac.il (128.139.210.16)            5 ms   5 ms   5 ms
3 128.139.198.129 (128.139.198.129)              9 ms  11 ms  13 ms
4 TAU-shiber.route.ibm.net.il (192.115.73.5)   535 ms 549 ms 513 ms
5 fe7507.tlv.ibm.net.il (192.116.177.1)        562 ms 596 ms 600 ms
6 165.87.220.18 (165.87.220.18)               1195 ms1204 ms
7 nyc28-16-sar1.ny.us.ibm.net (165.87.28.19)  1208 ms1216 ms1233 ms
8 198.133.27.5 (198.133.27.5)                 1210 ms1239 ms1211 ms
9 sprint-nap.bbnplanet.net (192.157.69.51)    1069 ms1087 ms1122 ms
10 nyc1-br2.bbnplanet.net (4.0.1.25)           1064 ms1109 ms1061 ms
11 cambridge1-br1.bbnplanet.net (4.0.1.122)    1185 ms1146 ms1203 ms
12 cambridge2-br2.bbnplanet.net (4.0.2.26)     1185 ms1159 ms1073 ms
13 ihtfp.mit.edu (192.233.33.3)                1052 ms 642 ms 658 ms
14 W20-RTR-FDDI.MIT.EDU (18.168.0.8)            640 ms 665 ms 674 ms
15 DANDELION-PATCH.MIT.EDU (18.181.0.31)        702 ms 915 ms 868 ms

The 3rd hop takes the path to the academic network [checked by local whois lookup], the fifth hop takes the path to New-York [on the east coast], and the 10th hop takes the path to Cambridge [in Massachusetts, on the coast, northern to New-York].

There is a utility named VisualRoute (http://www.visualware.com/visualroute/index.html) which traceroutes a host, and displays the route on a map of the world. The host's location on the map is based on the whois query, which may be wrong - an Israely domain might be displayed as being in Israel though it is hosted in another country.

Some of the services available on the host might give further info.


E.g. telnet construct.haifa.ac.il 13  <== Time of day service
  Trying 132.74.18.2...
  Connected to construct.haifa.ac.il.
  Escape character is '^]'.
  Wed Jan 21 08:32:53 1998         <== Time difference hints at the
                                       host's time zone.

Naming conventions of ISPs and back-bones

AT&T dialups : <port>.<router-location>.<state>.dial-access.att.net

Port is 2-254 for the dial-up ports, and 1 for the router itself. location: example: "los-angeles-2" (city and router #). state: 2-letter abbreviation.

uu.net dialups :

A. <port>.<device>.<city>.<state>.<iu>.uu.net

B. <port>.<device>.<airport>.<iu>.uu.net

iu = intended use (meaningless), state is per USPS ZIP code, deviceis Ascend 'TNT' # or Ascend 'MAX' #.

W3C DOM -Introduction

2007-10-24T03:43:00.001-07:00

The Document Object Model (DOM) is the model that describes how all elements in an HTML page, like input fields, images, paragraphs etc., are related to the topmost structure: the document itself. By calling the element by its proper DOM name, we can influence it.

On this page I give an introduction to the W3C Level 1 DOM that has been implemented in the newest generation of browsers. It will give you an overview of how the DOM works and what you can do with it.

First some words about the DOM Recommendation and the purpose of the DOM, then I teach you what nodes are and how you can walk the DOM tree from node to node. Then it's time to learn how to get a specific node and how to change its value or attributes. Finally, I'll teach you how to create nodes yourself, the ultimate purpose of the new DOM.

The Recommendation

The Level 1 DOM Recommendation has been developed by the W3C to provide any programming language with access to each part of an XML document. As long as you use the methods and properties that are part of the recommendation, it doesn't matter if you parse an XML document with VBScript, Perl or JavaScript. In each language you can read out whatever you like and make changes to the XML document itself.

As some of you might have guessed, this paragraph describes an ideal situation and differences (between browsers, for instance) do exist. Generally, however, they're far smaller than usual so that learning to use the W3C DOM in JavaScript will help you to learn using it in another programming language.

In a way HTML pages can be considered as XML documents. Therefore the Level 1 DOM will work fine on an HTML document, as long as the browser can handle the necessary scripts.

You can read out the text and attributes of every HTML tag in your document, you can delete tags and their content, you can even create new tags and insert them into the document so that you can really rewrite your pages on the fly, without a trip back to the server.

Because it is developed to offer access to and change every aspect of XML documents, the DOM has many possibilities that the average web developer will never need. For instance, you can use it to edit the comments in your HTML document, but I don't see any reason why you would want to do so. Similarly, there are sections of the DOM that deal with the DTD/Doctype, with DocumentFragments (tiny bits of a document) or the enigmatic CDATA. You won't need these parts of the DOM in your HTML pages, so I ignore them and concentrate instead on the things that you'll need in your daily work.

Nodes

The DOM is a Document Object Model, a model of how the various objects of a document are related to each other. In the Level 1 DOM, each object, whatever it may be exactly, is a Node. So if you do

 
This is a paragraph

you have created two nodes: an element P and a text node with content 'This is a paragraph'. The text node is inside the element, so it is considered a child node of the element. Conversely, the element is considered the parent node of the text node.

               <-- element node
              |
              |
    This is a paragraph <-- text node

If you do

This is a paragraph

the element node P has two children, one of which has a child of its own:

              
              |
        --------------
       |              |
   This is a         
                      |
                      |
                   paragraph

Finally there are attribute nodes. (Confusingly, they are not counted as children of an element node. In fact, while writing this pages I've done a few tests that seem to indicate that Explorer 5 on Windows doesn't see attributes as nodes at all.) So

This is a paragraph

would give something like

               ----------------
              |                  |
        --------------         ALIGN
       |              |          |
   This is a                 |
                      |        right
                      |
                   paragraph

So these are element nodes, text nodes and attribute nodes. They constitute about 99% of the content of an HTML page and you'll usually busy yourself with manipulating them. There are more kinds of nodes, but I skip them for the moment.

As you'll understand, the element node P also has its own parent, this is usually the document, sometimes another element like a DIV. So the whole HTML document can be seen as a tree consisting of a lot of nodes, most of them having child nodes (and these, too, can have children).

                          |
              |-------------------------------------
              |                                     |
              ----------------            lots more nodes
              |                  |
        --------------         ALIGN
       |              |          |
   This is a                  |
                      |        right
                      |
                   paragraph

Walking through the DOM tree

Knowing the exact structure of the DOM tree, you can walk through it in search of the element you want to influence. For instance, assume the element node P has been stored in the variable x (later on I'll explain how you do this). Then if we want to access the BODY we do

x.parentNode

We take the parent node of x and do something with it. To reach the B node:

x.childNodes[1]

childNodes is an array that contains all children of the node x. Of course numbering starts at zero, so childNodes[0] is the text node 'This is a' and childNodes[1] is the element node B.

Two special cases: x.firstChild accesses the first child of x (the text node), while x.lastChild accesses the last child of x (the element node B).

So supposing the P is the first child of the body, which in turn is the first child of the document, you can reach the element node B by either of these commands:

document.firstChild.firstChild.lastChild;
document.firstChild.childNodes[0].lastChild;
document.firstChild.childNodes[0].childNodes[1];
etc.

or even (though it's a bit silly)

document.firstChild.childNodes[0].parentNode.firstChild.childNodes[1];

Getting an element

However, walking through the document in this way is quite cumbersome. You'll need to be absolutely certain of the structure of the entire DOM tree and since the whole purpose of the Level 1 DOM is to allow you to modify the DOM tree, this could lead to problems really quickly.

Therefore there are several ways of jumping directly to an element of your choice. Once you have arrived there, you can walk the last bit of the DOM tree to where you want to be.

So let's continue with our example. You want to access the element node B. The very simplest way is to directly jump to it. By the method document.getElementsByTagName you can construct an array of all tags B in the document and then go to one of them. Let's assume that this B is the first one in the document, then you can simply do

var x = document.getElementsByTagName('B')[0]

and x contains the element node B. First you order the browser to get all elements B in the document (document.getElementsByTagName('B')), then you select the first of all B's in the document ([0]) and you've arrived where you want to be.

You could also do

var x = document.getElementsByTagName('P')[0].lastChild;

Now you go to the first paragraph in the document (we assume that our P is the first one) and then go to its lastChild.

The best way, the only way to be certain that you reach the correct element regardless of the current structure of the DOM tree, is to give the B an ID:

This is a paragraph

Now you can simply say

var x = document.getElementById('hereweare');

and the element node B is stored in x.

Changing a node

Now that we have reached the node, we want to change something. Suppose we want to change the bold text to 'bold bit of text'. We then have to access the correct node and change its nodeValue. Now the correct node in this case is not the element node B but its child text node: we want to change the text, not the element. So we simply do

document.getElementById('hereweare').firstChild.nodeValue='bold bit of text';

and the node changes. Try it and change it back again .

This is a paragraph

You can change the nodeValue of each text node or each attribute. Thus you can also change the ALIGN attribute of the paragraph. Try it and change it back again .

This, too, is quite simple. Take the node you need (the B's parentNode, in this case), then use the setAttribute() method to set the ALIGN attribute to the value you want:

function test2(val) {
 if (document.getElementById && document.createElement)
{
  node = document.getElementById('hereweare').parentNode;
  node.setAttribute('align',val);
}
 else alert('Your browser doesn\'t support the Level 1 DOM');
}

Creating and removing nodes

Changing nodes is nice, it can even be useful, but it's nothing compared to actually creating your own nodes and inserting them into your document. I can easily add an HR right below this paragraph and remove it quite as easily.

Creating the element is done by a special method:

var x = document.createElement('HR');

Thus an HR is created and temporarily stored in x. The second step is to insert x into the document. I wrote a special SPAN with ID="inserthrhere" at point where it should appear. So we use the appendChild() method on the SPAN and the HR is made a child of the SPAN and it magically appears:

document.getElementById('inserthrhere').appendChild(x);

Removing it is slightly more complex. First I create a temporary variable node to store the SPAN, then I tell it to remove its first child (the HR).

var node = document.getElementById('inserthrhere')
node.removeChild(node.childNodes[0]);

In the same way we can create a new text node and append it to our faithful B ID="hereweare"

var x = document.createTextNode(' A new text node has been appended!');
document.getElementById('hereweare').appendChild(x);

Try it , then go up to see the result. You will notice that executing the old functions does not remove the new text node, that's because it has become a separate node:

           
           |
      ------------
     |            |
 paragraph    A new text node
              has been appended!

(To merge them into one node use the normalize() method that's sadly not supported by Explorer 5 on Windows).

I won't tell you how to remove the text node, try writing that script yourself. It'll be a useful exercise.

Web 2.0 and the AJAX

2007-10-24T03:40:00.000-07:00

Web 2.0 is a strange thing in that it doesn't really exist. You can't buy Web 2.0; you can't buy a Web 2.0 programming language, and you can't buy Web 2.0 hardware. In many ways, the phrase "Web 2.0" is a marketing phrase like "paradigm shift" or "the big picture". The reason for this vagueness is that Web 2.0 doesn't have a tightly defined definition. What the phrase Web 2.0 tries to express is, that modern websites are so much better than early websites that they'd better be given a different name. So it is down to marketing.

Web developers need to demonstrate that they may use the same Internet, the same web browsers and the same web servers as their competitors, yet their websites are in fact an order of magnitude better. “Our competitors only do websites. We do Web 2.0 websites!"

The client is, of course, hugely impressed that his new website will be a Web 2.0 website. But what should he expect to see for his money? What is the client's view of what Web 2.0 should offer? Is it all smelling of roses or are there some thorny issues too?

I propose that there are in fact three facets to a Web 2.0 website:

1. AJAX

2. Social Networking (Building Communities)

3. Broadband

AJAX is technical and can only be performed by a technically skilled developer, social networking is vague, woolly and is based more on marketing models than web skills, and broadband has been popular for a long time. Even stranger is the fact that AJAX has been available to developers for at least 5 years, and social networking has been around even longer. It is simply the re-branding of these things that is causing the rise in the popularity of these old but current "buzzword" technologies.

AJAX is a mash up of technologies. We've had asynchronous JavaScript and XML for many years, but until somebody said "I name this mash up - AJAX" it remained out of the mainstream. The same goes with social networking. Forums, blogs, and community-based websites have been around for many years, but giving it a title like "social networking" combined with the success of websites such as www.Youtube.com and www.Linkedin.com makes it mainstream and popular. And to cap it all, the new names invented to re-brand existing technologies are combined into the all encompassing name of Web 2.0. Web 2.0 is simply rebranding the rebranded.

In summary, we've had the ability to create Web 2.0 websites for years. It is not new technology; it is simply the renaming and repackaging of something we already have and enjoy. Marketing has made buzzwords of what we already knew and the public and developers are lapping it up.

The third facet of Web 2.0 was broadband, or as I prefer to call it, broadband abuse. Many developers believe that Web 2.0 is defined by how long it takes to download a website or the size of the broadband connection required to view the site comfortably. They believe that the bigger the connection required or the longer the website takes to download, the more Web 2.0ish the website must be. In my opinion, however, adding vast images, video footage, badly implemented rounded corners and streaming music does not make a Web 2.0 website. It simply makes a regular website that is bloated and annoying.

Presuming that you understand what makes a Web 2.0 website and you are keen to build one, there is an important area that you should consider before you start. And that is the area of Search Engine Optimisation.

So what about search engines? Do Web 2.0 websites perform well on search engines? Do search engines need to change to keep pace with development? If we ignore the broadband abusers and look at the two key facets of Web 2.0, AJAX, and social networking we get two very different answers.

Working somewhat in reverse here, the conclusion is that AJAX is a search engine killer. Adding AJAX functionality to your website is like pulling the plug on your search engine strategy. Social networking sites on the other hand typically perform exceptionally well on search engines due to their vast amount of visitor provided content.

The reason AJAX is a search engine killer is pretty obvious once you know how the technology works, and at the risk of offending all the people who know this already, I'll recap in a brief paragraph.

Simply put, AJAX removes the need to refresh a page in a browser. Say for example, you are on the product-finding page of a website, you can type in a search phrase for the product you want to find and press the submit button. Without refreshing the page, the asynchronous JavaScript runs off, grabs the results of the search, and inserts the details of the found products into the very same page as you sit and look at it.

For the website user this addition of AJAX to the website feels fantastic. No page reloads, no browser flicker, no click noise, but sheer joy. And so the rush for AJAX websites begins, because the visitors will love it.

But what about the search engines, what will they make of web pages that use AJAX to find content? Importantly, search engines don't run JavaScript. Oh no, not ever, no way José. So the search engine will never run your AJAX. To the search engine, huge areas of your website content are now hidden, never to be spidered, indexed, or found. This really limits the usefulness of AJAX in many applications.

An ideal application of AJAX is Google Maps, where as you drag the map around the browser window, the newly exposed areas of the map are retrieved and shown on the page without a page refresh—smooth, seamless, and very impressive. Does Google care if the single map page gets found by searching? Certainly not!

A very poor application of AJAX is the product portfolio where you can find and view product details for hundreds of products without ever refreshing the page. Nice to use? Yes. Navigation friendly? No—try hitting the back button when the browser ignores your last 20 clicks because you have remained on the same page! Search engine friendly? Forget it. You are invisible.

So what is the solution to the AJAX invisibility cloak that Master Harry Potter himself would be proud of? There are 5 options:

Build two websites, one using AJAX that is lovely for visitors and another using more traditional techniques for search engine spiders to find. If you can find a client to finance both, you have found a client with too much money!
Drop AJAX. Let the visitors suffer the page refresh.
Run with AJAX anyway and just put up with the fact that your perfectly formed website will receive no search engine visitors.
Lobby the major search engines to rebuild their spidering algorithms to take into account AJAX pages and to run JavaScript on the pages they index. This option might take some time :-)
Increase your Google Ad words payments and ramp up traditional advertising to counteract the missing website traffic from the search engines.

And so, a bleak picture of AJAX is painted and by implication of Web 2.0 as well. The good applications of AJAX and Web 2.0 are few and far between, but when you do find them they are fantastic. Do you remember that feeling when you fist used Google Maps? Do you find that all other mapping websites now feel old fashioned? I would go as far as to say that it was Google Maps that single-handedly bought the technology of AJAX to the masses.

The second most impressive application of AJAX is another Google idea, where when typing in the search field on the Google website, AJAX is used to find results even as you type the words—incredibly quick to use, fantastic for the website visitor, and really demonstrating the technology in a great light.

Isn't it hugely ironic then that the one website that demonstrates so well the very technology that, if used on our own websites, will force us to spend more on Google Ad words, is in fact Google.