Is Enterprise Application Performance a Priority?

Yesterday I was in a meeting with several enterprise application developers, the users of our application were complaining that page load times had increased to 4 seconds. One of the users mentioned that google loads in under a second.

What was the general response? Well, this is pretty good, we're not google after all. If we had the resources of google, maybe we could get our page load times down.

To me 4 seconds is an eternity for a user application, on most of the apps I've worked on we've usually had response times less than 300ms. It is usually fairly easy to meet that goal with a combination of caching, sql tuning and using tools like jawr.

I usually don't like to release code with high page load times unless there is a good reason.

So my question is, Is Enterprise Application Perfomance a Priority?

Google Public DNS - Speed Up Application User Experience

Google Public DNS is primarily aimed at users of the internet. I had run across it surfing the web and filed it away. But it wasn't until my configured DNS servers for my local network went down (again) that I decided to give it a shot.

A couple of things that I notice right away is that the IP addresses are easy to remember:

8.8.8.8 and 8.8.4.4

That's actually something I am going to be able to recall next time I find myself with broken DNS services. The second thing is that my ping to these are 24ms roughly, that's not too bad. The real test is lookup speed, and this is where Google is riding on the edge as they appear to be doing some aggressive pre-lookup of DNS entries on slow DNS servers. Are they going to break TTL? A lot of people design fail-over using a lower TTL in their DNS responses so that the cache will be flushed if a failover is needed to a new IP address.

In any case, I've giving it a go for a while and so far my surfing does feel faster.

So if your public facing users are complaining about a slow web experience on your site, one suggestion is to point them to the free Google DNS servers to see if that improves their experience.

8 Real Ways to Save IT Costs

Times are tough, everyone is asked to reduce spending. Some IT departments are seen as an expendable cost center - if they are not saving money, they are not doing their jobs. I think this can be a healthy attitude to keep those sometimes-unaccountable IT departments honest and focused on the corporate business. I have a few high-tech ideas for cost savings that can yield real results. If I were the CIO of your company, this is what I would do to reduce costs in the data center and desktops:

1. Use commodity server hardware - do you really need that proprietary big iron Unix server? Wouldn't a Linux blade work instead? Intel x86 architecture can run faster and cheaper than that proprietary hardware, and you'll be surprised at the difference - you might get a 3X performance boost and cut your costs 75% at the same time! Unbelievable? I've seen it myself.

2. Use a commodity server OS - do you need Unix? Linux would work for 70% savings. If you are not truly married to Windows Server, switch that to Linux too, you're operations people will love it.

3. Use an Open Source app server - use JBoss, its better and cheaper than Oracle or IBM app servers. If you can re-write your Microsoft apps, do that too. Or just halt all future new Microsoft software projects in favor of Java / JBoss apps.

4. Use Open Source database servers - Postgres and MySQL have evolved in the last 5 years to be true powerhouse database engines. If you are using Oracle or IBM, you could save millions by switching your old apps and using Open Source databases for new apps. You may have a master license agreement that gives you "unlimited copies" of Oracle, well... how generous of you to line the pockets of the Oracle salesman. You'll find out that MySQL and Postgres are much easier to maintain too, so you will be able to reduce DBA costs at the same time. Want to get world-class support? Pay Sun for MySQL support or EnterpriseDB for Postgres support.

5. Outsource your email servers to Software as a Service (SaaS)- stop paying for Microsoft Exchange or IBM Notes, use Google Apps. If you don't like Google there are hundreds of alternatives, all with professional grade email and your customers and colleagues will never notice the change when they get emails from you. Pull the plug on your in-house email servers and reuse the hardware or turn them off to save electricity.

6. Outsource your whole data center - do you truly need to worry about having 30 servers in a concrete bunker with UPS, disaster recovery, HVAC, fast internet lines, and monitoring? Hire a hosting company that can do that better and cheaper than you ever could. You can host your test servers, your database servers and your internal web apps. Users won't even know that their apps are really running in a separate data center. You can get dedicated server hardware and fast links to their data center if you need it. Or save a huge amount on VPS services.

7. Use Open Source Office apps on the desktop - OpenOffice.org has truly arrived, with version 3.0 just released, it can read the new docx formats that Microsoft Office 2007 uses. Admittedly it does not behave exactly the same, but I used OpenOffice almost exclusively for Documents, Spreadsheets, and Presentations. I can easily send them to my colleagues who use Microsoft Office without compatibility problems. For email clients you can use Thunderbird to replace Outlook. Google Docs is making some waves but it is still immature and buggy. I look forward to seeing Google improve these apps enough to compete.

8. Use an Open Source OS on the desktop - Ubuntu and Fedora have really come far in the past few releases to offer truly solid Linux desktop experiences. Most Windows users will have no problem with the new UI. This may seem like a risky move, but if you are serious about cost-cutting, I think you'll find that Windows is a commodity in your office already. Now that email and office apps will run on Linux desktops, is Windows a requirement for your business? You may hit some web sites or Word documents that need Microsoft technologies, but I bet that's more rare than you think.

These are concrete, no-fluff ideas that any IT department can use. Obviously smaller, more agile IT departments and companies will find these changes easier to swallow. And some of these ideas might seem pretty risky, but if you want to keep risk down then try things out on new projects first. Later, switching the older systems to complete the cost savings. Its OK to mix Linux and Unix machines in the same data center, and there's no need to switch everything over night. You can get savings project by project to keep some sanity in the environment.

Making these changes won't be easy, many will resist these changes. These excuses are common:
* We already know how these old systems work, lets just buy more and keep it safe
* Open Source is risky, who can we blame for problems, what if we get sued?
* Nobody ever got fired for buying IBM (/Oracle)!

All of these are weak arguments to prevent the hassles involved with any change. Spending new money on old expensive solutions is wrong, and you can pay for professional support from RedHat, Sun, Canonical, and still come out way ahead on costs. At IBM and Oracle's prices, you should be fired for choosing them when lower cost competitors that would fit your business even better.

Cost saving pitfalls. I do not advocate cost-cutting in non-commodity areas of your business or IT. For example, Offshoring software development or customer service can have bad consequences. Quality varies wildly at home and abroad for software developers and customer service experts. Offshoring those essential skills just multiplies the risk by 6 or 10 time zones. I also would not expect the Sales department to outsource the sales department - company image and product knowledge are too important to the business to trust to others who are not invested in your future.

History shows us the way. I remember the switch from mainframes to Unix and Windows in the 90's economy. It did not happen overnight, and we heard plenty of grumblings from the luddites who resist all changes. It started with commodity apps like email and word processing (anyone remember the mainframe spell checker? yechkkk!), and one business application at a time to reduce the risk. We mixed mainframes and server apps for a decade, and we realized that we could reduce costs pretty quickly. As a side-effect we got more choice and better apps.

The same thing is happening in Open Source and the Software as a Service (SaaS) industry. Lower costs, more choices, better products and services. And it starts with commodities like operating systems, database engines, app servers, and apps like email and word processing. I use all of these technologies to save costs at my company.

Don't cling to expensive Microsoft, Oracle, and IBM products that haven't changed much in 10 years. Or you'll suffer the same fate as those mainframe-clingers in the 90's. Instead, use the latest technologies to save money and help your business in these trying times.


- Jay Meyer

Cluster your application - NOW!

Is your Java Application Server clustered? Why not? It should be, and your reasons for NOT clustering have vanished, so there's no excuse. Cluster. Now.

When designing a web architecture for an application, this question always comes up: "To cluster or not to cluster?" Clustering allows two or more app servers to act as one. This has many advantages, such as high availability if one server goes down, or the cluster can enable seamless upgrades to software or hardware. Clustering also allows a successful application to scale by spreading users across more and more servers. Unfortunately, clustered servers are much too rare today. But they shouldn't be rare, clustering should be the default, the normal case.

Question: To cluster or not to cluster?
At first the answer is "no", I mean... clustering make the installation more complex, there are performance impacts, and configuration problems and... well... RISK! But the benefits can be pretty great for an app that is growing. So if we can eliminate the RISK, then we have no reason NOT to cluster.

Answer: Always Cluster.
The time for clustering every Java application has come. Worrying about statefullness was a habit we acquired when SFSBs in EJB2 were dangerously bad at clustering (and a bad solution for many applications). But today, JBoss, Tomcat, and other Java app servers have matured to make clustering a solid part of any server. Terracotta has even come in to save the day for clusters with large node counts, or with huge memory needs, so the reasons not to cluster become smaller.

RedHat is telling everyone to virtualize Linux.. ALWAYS! And it makes sense. Servers out there need the flexibility of virtualization, and the reasons to NOT virtualize have vanished. It's time has come. Technologies like Xen and VMWare have improved, but more than the technologies, the people have matured to understand the benefits and risks of virtualization. So now it's packaging and marketing that are driving virtualization as the default answer.

Java application server clustering should now follow and become the default installation. The technologies are solid, so the clustering idea is just in need of good packaging such as easier installation and configuration utilities, and good marketing in the form of endorsements and documentation about the many benefits of clustering.

The benefits:

• High Availability - Server crashes, app stays up, 'nuf said, this bene' is obvious.
• Scalability - more users? add more servers, clustering lets your app grow - also an obvious bene'
• Hot Deployment - this bene' is rarely understood, but it will save your weekend! Stop deploying on a weekend at 2am! Deploy on Tuesday at 11am, yes, while users are still using the system.... You see, most code changes do not require huge data conversions and outages (although those cases do happen). Most deployments are to fix a bug, change a label, add a brand new feature, or add more memory to the server - and those changes do not require downtime if I have a cluster and a deployment process that can make it HOT. Its quite simple - 1)take a server node out of the cluster, 2) upgrade the code, 3) put it back in the cluster, 4)repeat on other nodes. No late nights, no weekends -and no lost sleep. Developers and admins are handy and awake if anything goes wrong, no need to start that late-night conference call. Freedom...

Believe it? You may not, but I've witnessed it. My system took a millions of hits a day, and we deployed twice a day at times - between 8 and 5. We had a cluster of JBoss application servers and did the deployments in broad daylight without any affect on users. And all this with out-of-the-box clustering from JBoss.

If you want to talk RISK - try taking an outage at 2am on a Sunday for 5 hours after 3 months of intense development by a large team. Put in a massive code change and pray that the deployment goes well... Now that's the stuff that nightmares are made of.

So cluster your Java servers, the benefits are big, and clustering comes with every application server - it probably came with the server you've got. No excuses, do it now.

-Jay Meyer

Stealth mode: upgrade your system and don't tell everyone

Want to try JPA or Spring? But you have an old software system, and don't have the luxury of a clean new software project?

Don't wait!
Add Spring and/or Hibernate to your old Java software system. Instead of waiting, put in a STEALTH upgrade that is low-friction, low cost, and low visibility to the customers and others around the project. You can boast later when the system succeeds because of the new technologies.

We've all made some excuses for NOT upgrading the technology of a system. (But we can address each of these):
* I can't afford to overhaul the whole system
* Our Java servers don't support it, we cannot upgrade the Java server
* What will my manager or customer think?

Upgrade the whole system?
No software engineer has time & money to do that. So I suggest looking for a new feature and only upgrading that part of the system: this is the low-friction part. There's no need to overhaul every working DAO, leave them for later until you need to change them to keep costs down. Maybe you've got a new model object you are adding, why not use Spring and Hibernate to manage that object? Who needs to drag around the old EJB2 patterns, or your own home-grown JDBC DAO's just to keep the system consistent? In the end, you may finish the feature upgrade faster because you used Hibernate and Spring Transactions. After you try it, you may decide to convert other DAO's as changes happen to those parts of the system. This can all be done under the radar - Stealth mode - without kicking off a major overhaul of every DAO in the system.

Upgrade my app server?
You may think that your Java App server doesn't support Hibernate or Spring, but Hibernate and Spring are simply jar libraries that can be deployed inside your web app, and do not require an upgrade to your server. So go ahead and add the jar files inside your web app (or WAR or EAR file), and use them freely. If you'd like to use EJB3, and you have an app server that does not support it like Tomcat or IBM, you could use JBoss Embeddable EJB3 container- yet another free, OSS jar library that can be put inside your web app. Hibernate and Spring can even work in Java 1.4, (EJB3 and JPA need Java5).

What will people think?
Your customers and users of the system may not be able to spell JDBC, but that's OK, besides that is why they pay YOU, the software engineer. So they will not care whether you use Hibernate, OpenJPA, iBATIS, or raw JDBC. Many of your managers will not care either, as long as it doesn't cause the boat to rock in server operations (and it won't as described above in upgrading your app server). If your customer/manager is curious, you can tell him that many good things will be coming: like short development times, developers will be excited about the new technology. If you need to convince other developers, remind them that these technologies are popular for a reason, and its a good skill for a software developer to have for the future.

Marketing this kind of strategy is a challenge. Selling this idea to your customers, colleagues, and managers is probably a bigger challenge than actually making the software changes. Just keep in mind that some people will not care, and don't need to be scared by change, so leave them out of the conversation except to say that their new features will be coming soon. NOT telling them is not deception or evil, its just too much information (TMI), and it can sometimes be scary. So if they keep asking questions, tell them everything, but remember to tell them that confidence is high, risk is low, and everyone is doing it. The software people who know the system should get excited, not scared, by the new technologies adding power and speed to the system. Consider the alternative - try to hire a Java developer by telling him he gets to maintain old EJB2 CMP. Software engineers run away from those technologies, you won't get anyone to hire on without lying, and your current developers will leave in short order too.

The Stealth approach can avoid the hoopla and resistance that happens when any kind of change is suggested. So take advantage of the trust they place in you to make the right technology choice. You will get to use new technologies, your system will be better, and nobody needs to be the wiser. In the end you may get it all done faster and easier too.

-Jay Meyer

How to avoid SQL injection in Hibernate (A Hibernate Urban Legend)

Somewhere along the line java developers came to believe that Hibernate protects you from SQL injection. I'm not sure where they came to believe that. Maybe it is because you no longer have to write SQL and Hibernate does many other magical things - it has to protect you against SQL injection.

I'm tired of telling java developers that HQL has the same vulnerabilities as SQL, they don't believe me and think Hibernate offers them some sort of magical protection from bad HQL. Basically, what I term bad HQL is when named parameters are not used. Consider the following example:

String goodParameter="Raj lane";

Query badQuery = session.createQuery("from Address a where a.street='"+goodParameter+"'");

I have SQL logging turned on so I can see that the generated SQL is as follows:

select address0_.addressId as addressId, address0_.street as street1_ from Address address0_ where address0_.street='Raj lane'

Now consider the following where I attempt "HQL Injection"

String badParameter="la' or '1'='1";

Query reallyBadQuery = session.createQuery("from Address a where a.street='"+badParameter+"'");

And the resulting SQL:

select address0_.addressId as addressId, address0_.street as street1_ from Address address0_ where address0_.street='la' or '1'='1'

Note that the above SQL passes the parameter directly into the SQL. The generated SQL will return all rows in the table. Which is bad, but SQL injection opens us up to much worse attacks. So the moral of the story is to use named parameters, the above code can be fixed as follows:

String badParameter="la' or '1'='1";

Query reallyBadQuery = session.createQuery("from Address a where a.street=:street");

reallyBadQuery.setParameter("street", badParameter);

Rajesh Patel

Harpoon Technologies

Where Did I Put My Encryption Keys?

As a software security consultant, I was asked to solve some interesting problems with cryptography in Java. I warn others that security is esoteric and an afterthought much of the time. So taking security into consideration early is better. But even if you are a late-comer, the solutions do not cost that much, especially when compared to losing valuable data to an unfriendly attacker. Most of them can be done with Java packages that come with the JDK (the .NET framework has them too).

For example, a super easy solution to a common problem is password hashing. Passwords need to be hidden from attackers, programmers, admins or DBA's. This can be easily done with the MessageDigest class in Java. The great thing about hashing is that there is no key to store, you simply run your password through the algorithm, like SHA, and out pops a byte array that you can store in the database or a text file. Then the next time you prompt for the password, you hash that password with the same MessageDigest function, compared it to the one you've got stored, and if they match, you are logged in. Attackers or admins cannot possibly guess what that hash is by looking at it. So there you have a 3-line maintenance-free security solution that will pass any security audit - it's the same technique most Unix systems use to store their passwords.

Software cryptography provides quite a few different solutions, and when you analyze them, you try to find the weak spot. The weak spot is usually the keys to a cipher like 3DES or AES. When you use a cipher, you have a key, and storing this key safely can be a big problem since you don't want them to be stolen or
compromised. Most security books point this out and then fall short of suggesting some solutions: e.g. this one from Microsoft Press tells you to put the key somewhere far away from the data, but the example code violates that very suggestion by putting the key and the data right next to each other in the Windows registry (with a disclaimer that says don't do this). So what hope does a novice software developer have to avoid making bad choices when storing the keys? Other books on secure software are no better, I think their publishers forbid them to give good examples for fear of being sued.

Bad practices happen, but I will suggest some better ones since I have no publisher to forbid it. Let's define some security requirements for cryptographic keys that every developer should follow:
* The keys should be hidden
* The keys should not be known to the programmer
* The keys should be changeable after installation in case the keys are lost to an attacker
That's all we need, really. Now lets try to meet these requirements.

Solution 1: Hard-coding the keys in the code. Many developers will create a static String in the code that has encryption keys in it. This gets compiled and shipped with the software. Since the customer normally gets binaries and not source code, this hides the keys pretty well to satisfy the requirement above. However, the next two requirements are impossible with this solution. Now the programmer knows the keys - imagine being kidnapped by spies and tortured for the keys, or more likely, imagine being bribed for the keys. Then imagine what happens if someone bribes a programmer and then posts the keys on a public web site. Even easier than bribery, an attacker might download the software from your own web site, decompile it (amazingly simple for Java or C#), and then post your keys to a web site. You must panic, update the code, and every one of your customers must upgrade ASAP. This is a nightmare. You might say, "Yes, but most people would not know how to decompile the code, right?" Remember, you are not protecting your keys from MOST people, you are protecting them from a motivated attacker, and motivated people DO know how to decompile your code. (Yes, of course I know how to do it, but I won't bother putting the details here.)

Solution 2: Putting the keys in a config file. Storing your keys in a config file seems unpleasant because the keys are right there in plain text for all to read. Clearly this violates the hidden requirement above, doesn't it? Its true - its not hidden very well. However, the config file DOES satisfy the other two requirements: the keys are hidden from the programmer and they are changeable later. So now when the keys are lost to an attacker who posts them on a web site, you tell the customer to change the config file to enter new keys. The other customers need not worry, and the customer who lost their keys will be secure again in seconds.

Well maybe we can encrypt the config file! Yes, then we solve the hidden requirement! Brilliant! Except for one thing - where do you store the key for THAT encryption? Hard coded in the app? In another config file? You can see where this is going... The truth about security solutions is that they can all be beaten, some solutions are simply better than others and cause the attacker to expend more effort. In the case of the keys in the config file, you can simply use Unix or Windows file permissions to limit the readability of the file to the people who need to know, thus limiting the attackers who can get the keys. Then if the keys end up on a hacker web site, you change the keys, then haul off the admins to jail and torture them - but at least the programmers are safe and your customers are safe.

Some companies actually have a security policy that says "no storing passwords in plain text on the disk". I would say that this rule is impossible to follow. Because when I encrypt the password, where would I store the keys for that? Its a rule made by people who haven't thought about it. Sometimes this rule leads to hard-coding the keys in the code, and that's even worse. But if you have to follow this silly rule, use ROT13 since it needs no keys itself.

You can further protect the keys by putting them somewhere else separate from the data - maybe store them in a JNDI server instead of a config file (note that JNDI would persist to a text file, but at least it would be on a separate machine), or in a separate database from the rest of the data. (BTW: where did you store your database password? I suggest JNDI data sources, follow Solution 2 above). Anything to limit the exposure of the keys so you can hide them from most, but not all.

You can see that hard-coding the keys is appealing because it hides the keys, and what's more important than that? But hard coding the keys totally ignores the other two concerns, and it doesn't really do a great job of hiding the keys. So storing the keys in a config file (or other external place), even in plain text, is always preferred.

-Jay Meyer

jmeyer at harpoontech dot com