tag:blogger.com,1999:blog-70309973860281002042024-02-08T07:02:40.554-08:00Harpoon TechnologiesRajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.comBlogger23125tag:blogger.com,1999:blog-7030997386028100204.post-753550682732016372013-07-15T22:33:00.000-07:002013-07-15T22:33:19.347-07:00Strange characters in your Web apps? It could be Oracle's Unicode handling.Do you have strange characters in your Web app? Like a reversed question mark? Or characters like these: ⁿ‼↨♫☼◙ Sometimes it happens where you should see an apostrophe or a quote character like : Daniel◙s Bar-B-Q .<br />
<br />
Here's one reason why: <b><u>Oracle and Unicode</u></b>. My scenario went like this:
<ol>
<li> A user has a Document with some Unicode characters on it like the fancy "smart quote" characters (a Unicode u2019 for apostrophe) </li>
<li> The user does a copy & paste to put the text into the Web app's comments field (a Java String) </li>
<li> The web app saves the String into an Oracle VARCHAR2(4000) column called comments </li>
<li> The Oracle instance has a popular character setting: US-ASCII-7, which does not support Unicode characters </li>
<li> Oracle converts the character by truncating the leftmost bits, rendering an unprintable character ASCII(19) into the comments column </li>
<li> Later, the Webapp reads the Oracle column, displays it to the browser, and displays the unprintable ASCII(19) character. as something Strange like ♫ or ◙.
</ol>
<br/>
<b><u>Truncate Bits!!! What the...? WHY?? </u></b>
<br/>
<a href="http://www.youtube.com/watch?v=NNOa5Uvjpwo">Yeah, that's right.</a> Oracle truncated the bits, losing the original character - silently killing those Unicode characters <a href="http://www.youtube.com/watch?v=U74s8nFE7No">without so much as a "by your leave" </a>. Isn't "character assassination" illegal? How could Oracle DO such a thing?! The answer is not a simple one. First you need to <a href="http://www.joelonsoftware.com/articles/Unicode.html">understand Unicode character sets</a>, and then realize that Oracle has <a href="http://docs.oracle.com/cd/B10500_01/server.920/a96529/ch2.htm">tons of options and support for different character sets.</a> But we can imagine what Oracle designers were thinking when converting a big 16-bit character into a tiny 7-bit space:
<ul>
<li><b>Throw an error.</b> This would rollback transactions and force developers to handle or prevent the errors - pbbbt... what else you got?</li>
<li><b>Remove the Unicode character(s).</b> That would change the size of the value stored.</li>
<li><b>Convert the character to something else, quietly.</b> This is actually what happens, but it has it's own pitfalls: silent but deadly... pitfalls</li>
</ul>
Oracle can actually do some impressive up-converts : like from <a href="https://en.wikipedia.org/wiki/Windows-1252">Windows-1252</a> to UTF-8. But unfortunately, the down-convert smashing of "smart quote" chars into ASCII uses the most brain-dead algorithm - truncating the left most bits - and u2019 becomes x19, which is unprintable. And if you try to convert that to XML, x19 is not even a valid XML character and the parser crashes.
<br/><br/>
<b><u>Why Unicode? What are Smart Quotes?</u></b>
<br/>So how did these Unicode characters get into my App anyhow? "My app only uses English, it's a simple app", you might be saying. The answer is that popular apps like Microsoft Office will <a href="http://office.microsoft.com/en-us/word-help/change-curly-quotes-to-straight-quotes-and-vice-versa-HA010173242.aspx"> automatically change your " or ' characters into more fancy, curly quote characters as you type</a>. Nice, eh? Why not make your text look better? Who wouldn't want that? So now millions of MS Word documents with Unicode chars are floating around the planet, even your users are typing them, and they are copy & pasting those strings into your app. So THAT's where the Unicode came from for my scenario above.
<br/>And truly this should be OK, because tons of systems support Unicode already, Java and other languages use Unicode for their Strings internally already. And lots of apps like your Web browser support Unicode too. Oracle supports Unicode as well, but you do have to be aware at installation time. Oracle's US7ASCII setting is very popular, including on my Oracle instance, but I wouldn't recommend it. In today's world, you need Unicode, UTF-8 is well supported, and would have helped me here.
<br/><br/>
<b><u>Now What? How do I Fix It?</u></b>
<br/>You have a few options:
<br/><br/><b>1. Long-term, switch to UTF-8.</b> UTF-8 is well-supported and it's a super-set of ASCII, so all of the old ASCII text should convert in all of your VARCHAR2 columns. For almost all characters, the size is still just 8-bits, but can be larger for smart-quotes and others. However, converting an entire Oracle instance is nuclear - if you have a large database, this can take time, and the <a href="http://docs.oracle.com/cd/B19306_01/server.102/b14225/ch11charsetmig.htm">conversion your DBA recommends</a> may be akin to backing up everything, and re-importing all of your data: hours or days of effort.
<br/><br/><b>2. Use NVARCHAR2.</b> You might consider the Oracle NVARCHAR2 column type just for your large comment-style fields that need free-form text. This will only require an ALTER TABLE on that column. But you will have to remember this for new Tables with similar needs.
<br><br/><b>3. Use a BLOB. </b> If you store the field in a BLOB column, Oracle will not do any conversion. For columns where you don't care about indexes and search, like a comments section on your app, this should be OK. In Java, for example, you may not have to change any code at all. This is because you can still directly use this column as a String, i.e. JDBC Statement.setString() and getString() methods work fine without the <a href="http://docs.oracle.com/javase/6/docs/api/java/sql/Blob.html">messy JDBC BLOB handling</a>. Or, JPA's @Lob annotation can also be placed on a String for minimum coding effort. As an added benefit, your users can paste the entire contents of "War and Peace" without having a size limit of 4000 in VARCHAR2. Note: Oracle's CLOB column type will still do character conversion, so it will not help here.
<br><br/><b>4. Filter the characters. </b> In the short term I ended up <a href="http://stackoverflow.com/questions/397250/unicode-regex-invalid-xml-characters">using a regex to filter out </a> and remove any unprintable characters. Unfortunately this lost the character, turning "Dan's Bar-B-Q" into "Dans Bar-B-Q". Users were not too happy with that, but it did stop the bleeding until I can implement one of the above solutions.
<br/><br/>
<b><u>Conclusion, Recommendation</u></b>
<br/>If you have strange characters appearing in your application, check the character set in your database engine. Oracle, SQL Server, Postgres, MySQL - they can all support a wide range of settings. Also, think about the design of your fields, do you really want VARCHAR2(4000) for a comments field? Maybe BLOB would be better?
<ul>
<li>Use UTF-8 in your database engine</li>
<li>Use a BLOB (careful, not CLOB)for a comments column. You'll get a big size, and still use a simple String in memory</li>
<li>Make a separate table for all of the comment fields, you probably have many comment fields in your app:</l1>
<pre>
CREATE TABLE COMMENT(
ID VARCHAR2(20) PRIMARY KEY,
COMMENT_TEXT BLOB
)
</pre>
</li>
</ul>
<br/><br/>
<b><u>Other Solutions</u></b>
<br/>
<a href="https://en.wikipedia.org/wiki/Windows-1252">Windows-1252</a>. This Microsoft-created character set looks attractive because it is still 8-bit, and it includes the curvy smart-quote characters and many other useful punctuation characters. Oracle supports it too, calling it WE8MSWIN1252 in their settings. However, this is really just a stop-gap, a hack, until good Unicode support arrived. So I cannot recommend this unless you are desperate. Technically it would work, but now that UTF-8 support is so good everywhere, and with disk space at pennies per GB, there's no need for it, and IMHO, Windows-1252 character set support should start to fade. Jay Meyerhttp://www.blogger.com/profile/02158043537042803017noreply@blogger.com3tag:blogger.com,1999:blog-7030997386028100204.post-44303394464604142622013-05-27T17:26:00.000-07:002013-05-27T18:34:19.065-07:00Using Oracle Stored Procedures and Java ResultSetOracle stored procedures are not my favorite slice of technology, but I have to use them sometimes. In this case I had a stored procedure that I needed to call: it returns a set of rows - a result set. But Oracle Stored procedures do not return anything by design - other database systems like SQLServer and Postgres can, but not Oracle. So what's a <a href="http://www.youtube.com/watch?v=OtAUsVXB9OU">Git-R-Done</a> <a href="http://pragprog.com/book/tpp/the-pragmatic-programmer">pragmatic developer</a> to do?
<br />
<br />
The stored procedure in question used an OUT parameter with a "cursor" type, this is how Oracle can "return" the results of a select statement. (<i>I can feel my CS professor grading this now: "D-, <u>NEVER</u> use Out Params!"</i> some languages like Java do not even have out params). And then the NEXT challenge: How do you call that from Java? Good question - this case doesn't happen in the abridged JDBC docs, nor in the Hibernate nor Spring frameworks docs either.
<br />
<br />
JDBC has standard support for ResultSets returned from a stored procedure CallableStatment, this is how many DB vendors like Postgres and SQLServer work. In those drivers, <a href="http://docs.oracle.com/javase/6/docs/api/java/sql/Statement.html">Statement.exeecuteQuery()</a> returns a ResultSet. But Oracle has a long history of having <a href="http://www.oracle.com/technetwork/database/enterprise-edition/jdbc-faq-090281.html">backwards</a> and low-compatibility issues for JDBC details,and this is another case of Oracle jamming their square-peg stored procedures into the round hole of JDBC. But you <b>can</b> make it work.<br />
<br />
Here are the "challenges":<br />
* Oracle Stored Procedures <a href="http://docs.oracle.com/cd/B19306_01/server.102/b14200/statements_6009.htm">CANNOT return a result set</a> (it's an old design?) <br />
* Oracle "gives" you <a href="http://docs.oracle.com/cd/B28359_01/appdev.111/b28370/static.htm#CIHCEICB">SYS_REFCURSOR</a> on an OUT param (not a JDBC standard, so that's an odd feeling) <br />
* Oracle's JDBC driver does not have a JDBC-standard way to do this, they require OracleType.CURSOR <br />
* Oracle's driver rejects the "OTHER" SQL Type, so you cannot use this JDBC Standard either
<br />
<br />
<h3>
Step 1: Your stored procedure </h3>
Oracle stored procedures can use an OUT param with type SYS_REFCURSOR to return results of a SQL Select:
<br />
<pre> CREATE PROCEDURE "JAY_PROC" ( st_cursor OUT SYS_REFCURSOR ).... --details below
</pre>
<h3>
Step 2: CallableStatment </h3>
Of course, been there, done that - the ol' JDBC way to call stored procedures:
<br />
<pre>CallableStatement cs = connection.prepareCall("call jay_proc(?)");
</pre>
<h3>
Step 3: register the OUT param </h3>
Feeling squeemish? It's OK... we are almost finished ...you'll feel a small prick in your good programming sense:
<br />
<pre>cs.registerOutParameter(1, OracleType.CURSOR);
cs.execute();
</pre>
<h3>
Step 4: blink... cast the ResultSet ??!</h3>
err.. Yes, it's true, the object returned is actually a ResultSet, but you have to cast it:
<br />
<pre>ResultSet rs = (ResultSet) cs.getObject(1);
</pre>
<h3>
Step 5: Use the ResultSet as normal, JDBC calls </h3>
Whew! It's over.. Now we have what we wanted, a ResultSet:<br />
<pre>while( rs.next()){
</pre>
<h3>
Conclusion</h3>
Oracle does not directly support stored procedures that return ResultSet, but with these 3 or 4 steps, you can get the stored procedure with a cursor to work. I assume that Oracle's JDBC driver does the work of mapping their cursor object to a JDBC-compliant ResultSet, which is nice. But if I had a preference, I would wish for Oracle to support procedures that can return rows from a select. Then you could use the execute() method to return a ResultSet directly without the odd casting operation in step 4.
<br />
<br />
Ideally you would use a SQL Select statement and the PreparedStatement JDBC object, but I was saddled with an existing stored procedure in this case. I do not recommend using stored procedures for almost any use case, if you can avoid it. Instead I prefer to use Java's standard JPA as much as possible.
<br />
<br />
Here is the working code:<br />
<br />
<pre>package com.harpoontech.test.dao;
import java.sql.CallableStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
public class TestOracleStoredProcedure {
/**
* The SQL TYPE for an Oracle CURSOR in a Oracle Stored Procedure.
* This duplicates the OracleTypes.CURSOR, but with this constant
* we do not need to import com.oracle.* jars into this project.
* However this class is still 100% dependent on Oracle at runtime
* and cannot be unit tested without Oracle.
*/
private int ORACLE_CURSOR_TYPE=-10;
public static void main(String[] args) throws SQLException {
TestOracleStoredProcedure t = new TestOracleStoredProcedure();
t.test();
}
public void test() throws SQLException {
//let Spring create a Data source for the Oracle connections
DriverManagerDataSource ds = new DriverManagerDataSource();
String url = "jdbc:oracle:thin:@myhost:1525:DEVDB";
ds.setUrl(url);
ds.setUsername("scott");
ds.setPassword("tiger");
CallableStatement cs = ds.getConnection()
.prepareCall("call jay_proc(?)");
cs.registerOutParameter(1, ORACLE_CURSOR_TYPE);
cs.execute();
ResultSet rs = (ResultSet) cs.getObject(1);
while( rs.next()){
System.out.println("title:"+rs.getString(1)+" author:"
+rs.getString(2));
}
rs.close();
cs.close();
}
/*
--the Oracle 9+ stored proc with 1 OUT parameter for the results of a select:
CREATE OR REPLACE PROCEDURE "JAY_PROC" ( st_cursor OUT SYS_REFCURSOR )
is
BEGIN
OPEN st_cursor FOR
SELECT TITLE, AUTHOR, ISBN_ID
FROM books;
end JAY_PROC;
*/
}
</pre>
<h3>
Notes</h3>
* I used the Spring DriverManagerDataSource, but you could replace that with straight JDBC calls<br/><br/>
* Notice I used my own constant ORACLE_CURSOR_TYPE to avoid importing the oracle dependencies.<br/><br/>
* You can pass the ResultSet object to another framework like Spring-JDBC, if you like<br/><br/>
* I found most of this by using the debugger to show what Oracle drivers were really doing<br/><br/>Jay Meyerhttp://www.blogger.com/profile/02158043537042803017noreply@blogger.com8tag:blogger.com,1999:blog-7030997386028100204.post-51948905094484919592012-02-02T07:46:00.000-08:002012-02-02T07:51:31.921-08:00Seeking C# Developers in St. Louis MOWe are currently looking for C# developers in St. Louis MO with 5+ years of experience. They must have experience using C#, .NET 3.5, Oracle 10g and 11g.<br /><br />Harpoon Technologies is an established IT Services Firm in business since 2005. We offer competive benefits:<br /><ul><li>Health</li><li>Dental<br /></li><li>401(k) with match<br /></li><li>Life Insurance</li><li>Short/Long Term Disability </li></ul><p>Interested candidates can send resumes to rpatel@harpoontech.com</p>Rajesh Patelhttp://www.blogger.com/profile/17382341715789193656noreply@blogger.com0tag:blogger.com,1999:blog-7030997386028100204.post-431712223200059962011-04-20T08:18:00.000-07:002011-04-20T08:21:10.832-07:00Seeking Senior Java Web Developer for Long Term ContractHarpoon Technologies is seeking a Senior Java Developer with expertise in JSF, Hibernate, Spring, Web Services. <br />
<br />
Translate application storyboards, use cases and web-page concept designs into<br />
functional, dynamic web applications.<br />
<br />
Contract is long term at around 18 months with an expected end data at the end of 2012.<br />
<br />
Harpoon Technologies is an established IT Services Firm in business since 2005. We offer competive benefits:<br />
* Health<br />
* Dental<br />
* 401(k)<br />
* Life Insurance<br />
* Short/Long Term Disability<br />
<br />
Interested candidates can send resumes to rpatel@harpoontech.comRajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.com0tag:blogger.com,1999:blog-7030997386028100204.post-29183194036069842382010-12-29T13:36:00.000-08:002010-12-29T13:36:10.012-08:00Java DevelopersWe are currently looking for Java developers with 5+ years of experience. They must have expert knowledge of spring, hibernate, and java web development.Rajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.com0tag:blogger.com,1999:blog-7030997386028100204.post-54295010474480326842010-10-01T08:47:00.000-07:002010-10-01T13:00:19.920-07:00Is Enterprise Application Performance a Priority?Yesterday I was in a meeting with several enterprise application developers, the users of our application were complaining that page load times had increased to 4 seconds. One of the users mentioned that google loads in under a second.<br />
<br />
What was the general response? Well, this is pretty good, we're not google after all. If we had the resources of google, maybe we could get our page load times down.<br />
<br />
To me 4 seconds is an eternity for a user application, on most of the apps I've worked on we've usually had response times less than 300ms. It is usually fairly easy to meet that goal with a combination of caching, sql tuning and using tools like jawr.<br />
<br />
I usually don't like to release code with high page load times unless there is a good reason.<br />
<br />
So my question is, Is Enterprise Application Perfomance a Priority?Rajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.com2tag:blogger.com,1999:blog-7030997386028100204.post-83566914286484035012009-12-18T08:30:00.000-08:002009-12-23T13:04:19.223-08:00Google Public DNS - Speed Up Application User Experience<a href="http://code.google.com/speed/public-dns/">Google Public DNS</a> is primarily aimed at users of the internet. I had run across it surfing the web and filed it away. But it wasn't until my configured DNS servers for my local network went down (again) that I decided to give it a shot.<br /><br />A couple of things that I notice right away is that the IP addresses are easy to remember:<br /><br /><blockquote>8.8.8.8 and 8.8.4.4</blockquote><br /><br />That's actually something I am going to be able to recall next time I find myself with broken DNS services. The second thing is that my ping to these are 24ms roughly, that's not too bad. The real test is lookup speed, and this is where Google is riding on the edge as they appear to be doing some aggressive pre-lookup of DNS entries on slow DNS servers. Are they going to break TTL? A lot of people design fail-over using a lower TTL in their DNS responses so that the cache will be flushed if a failover is needed to a new IP address.<br /><br />In any case, I've giving it a go for a while and so far my surfing does feel faster.<br /><br />So if your public facing users are complaining about a slow web experience on your site, one suggestion is to point them to the free Google DNS servers to see if that improves their experience.Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-7030997386028100204.post-36166773336810967082008-12-08T04:27:00.000-08:002008-12-31T11:43:07.556-08:008 Real Ways to Save IT CostsTimes are tough, everyone is asked to reduce spending. Some IT departments are seen as an expendable cost center - if they are not saving money, they are not doing their jobs. I think this can be a healthy attitude to keep those sometimes-unaccountable IT departments honest and focused on the corporate business. I have a few high-tech ideas for cost savings that can yield real results. If I were the CIO of your company, this is what I would do to reduce costs in the data center and desktops:<br /><br /><span style="font-weight:bold;">1. Use commodity server hardware</span> - do you really need that proprietary big iron Unix server? Wouldn't a Linux blade work instead? Intel x86 architecture can run faster and cheaper than that proprietary hardware, and you'll be surprised at the difference - you might get a 3X performance boost and cut your costs 75% at the same time! Unbelievable? I've seen it myself.<br /><span style="font-weight:bold;"><br />2. Use a commodity server OS</span> - do you need Unix? Linux would work for 70% savings. If you are not truly married to Windows Server, switch that to Linux too, you're operations people will love it.<br /><br /><span style="font-weight:bold;">3. Use an Open Source app server</span> - use <a href="http://www.jboss.org/">JBoss</a>, its better and cheaper than <a href="http://www.oracle.com/appserver/weblogic/weblogic-suite.html">Oracle</a> or <a href="http://www.ibm.com/websphere/">IBM</a> app servers. If you can re-write your Microsoft apps, do that too. Or just halt all future new Microsoft software projects in favor of Java / JBoss apps.<br /><br /><span style="font-weight:bold;">4. Use Open Source database servers </span>- <a href="http://www.postgresql.org/">Postgres</a> and <a href="http://www.mysql.com/">MySQL</a> have evolved in the last 5 years to be true powerhouse database engines. If you are using Oracle or IBM, you could save millions by switching your old apps and using Open Source databases for new apps. You may have a master license agreement that gives you "unlimited copies" of Oracle, well... how generous of you to line the pockets of the Oracle salesman. You'll find out that MySQL and Postgres are much easier to maintain too, so you will be able to reduce DBA costs at the same time. Want to get world-class support? Pay <a href="http://www.sun.com/software/products/mysql/">Sun</a> for MySQL support or <a href="http://www.enterprisedb.com/">EnterpriseDB</a> for Postgres support.<br /><br /><span style="font-weight:bold;">5. Outsource your email servers to Software as a Service (SaaS)</span>- stop paying for Microsoft Exchange or IBM Notes, use <a href="https://www.google.com/a/">Google Apps</a>. If you don't like Google there are hundreds of alternatives, all with professional grade email and your customers and colleagues will never notice the change when they get emails from you. Pull the plug on your in-house email servers and reuse the hardware or turn them off to save electricity.<br /><br /><span style="font-weight:bold;">6. Outsource your whole data center</span> - do you truly need to worry about having 30 servers in a concrete bunker with UPS, disaster recovery, HVAC, fast internet lines, and monitoring? Hire a hosting company that can do that better and cheaper than you ever could. You can host your test servers, your database servers and your internal web apps. Users won't even know that their apps are really running in a separate data center. You can get dedicated server hardware and fast links to their data center if you need it. Or save a huge amount on VPS services.<br /><br /><span style="font-weight:bold;">7. Use Open Source Office apps on the desktop</span> - <a href="http://www.openoffice.org/">OpenOffice.org</a> has truly arrived, with version 3.0 just released, it can read the new docx formats that Microsoft Office 2007 uses. Admittedly it does not behave exactly the same, but I used OpenOffice almost exclusively for Documents, Spreadsheets, and Presentations. I can easily send them to my colleagues who use Microsoft Office without compatibility problems. For email clients you can use Thunderbird to replace Outlook. <a href="http://docs.google.com/">Google Docs</a> is making some waves but it is still immature and buggy. I look forward to seeing Google improve these apps enough to compete.<br /><br /><span style="font-weight:bold;">8. Use an Open Source OS on the desktop </span>- <a href="http://www.ubuntu.com/">Ubuntu</a> and <a href="http://fedoraproject.org/">Fedora</a> have really come far in the past few releases to offer truly solid Linux desktop experiences. Most Windows users will have no problem with the new UI. This may seem like a risky move, but if you are serious about cost-cutting, I think you'll find that Windows is a commodity in your office already. Now that email and office apps will run on Linux desktops, is Windows a requirement for your business? You may hit some web sites or Word documents that need Microsoft technologies, but I bet that's more rare than you think.<br /><br />These are concrete, no-fluff ideas that any IT department can use. Obviously smaller, more agile IT departments and companies will find these changes easier to swallow. And some of these ideas might seem pretty risky, but if you want to keep risk down then try things out on new projects first. Later, switching the older systems to complete the cost savings. Its OK to mix Linux and Unix machines in the same data center, and there's no need to switch everything over night. You can get savings project by project to keep some sanity in the environment.<br /><br />Making these changes won't be easy, many will resist these changes. These <span style="font-weight:bold;">excuses</span> are common:<br />* We already know how these old systems work, lets just buy more and keep it safe<br />* Open Source is risky, who can we blame for problems, what if we get sued?<br />* Nobody ever got fired for buying IBM (/Oracle)!<br /><br />All of these are <span style="font-weight:bold;">weak</span> arguments to prevent the hassles involved with any change. Spending new money on old expensive solutions is wrong, and you can pay for professional support from <a href="http://www.redhat.com/">RedHat</a>, Sun, <a href="http://www.canonical.com/">Canonical</a>, and still come out way ahead on costs. At IBM and Oracle's prices, you <span style="font-weight:bold;">should</span> be fired for choosing them when lower cost competitors that would fit your business even better.<br /><br /><span style="font-weight:bold;">Cost saving pitfalls.</span> I do not advocate cost-cutting in non-commodity areas of your business or IT. For example, Offshoring software development or customer service can have bad consequences. Quality varies wildly at home and abroad for software developers and customer service experts. Offshoring those essential skills just multiplies the risk by 6 or 10 time zones. I also would not expect the Sales department to outsource the sales department - company image and product knowledge are too important to the business to trust to others who are not invested in your future.<br /><br /><span style="font-weight:bold;">History shows us the way.</span> I remember the switch from mainframes to Unix and Windows in the 90's economy. It did not happen overnight, and we heard plenty of grumblings from the <a href="http://en.wikipedia.org/wiki/Luddite">luddites</a> who resist all changes. It started with commodity apps like email and word processing (anyone remember the mainframe spell checker? yechkkk!), and one business application at a time to reduce the risk. We mixed mainframes and server apps for a decade, and we realized that we could reduce costs pretty quickly. As a side-effect we got more choice and better apps. <br /><br />The same thing is happening in Open Source and the <a href="http://en.wikipedia.org/wiki/Software_as_a_service">Software as a Service</a> (SaaS) industry. Lower costs, more choices, better products and services. And it starts with commodities like operating systems, database engines, app servers, and apps like email and word processing. I use all of these technologies to save costs at my company.<br /><br />Don't cling to expensive Microsoft, Oracle, and IBM products that haven't changed much in 10 years. Or you'll suffer the same fate as those mainframe-clingers in the 90's. Instead, use the latest technologies to save money and help your business in these trying times.<br /><br /><span style="font-style:italic;"><br />- Jay Meyer</span>Jay Meyerhttp://www.blogger.com/profile/02158043537042803017noreply@blogger.com2tag:blogger.com,1999:blog-7030997386028100204.post-41658917928031078282008-10-19T19:55:00.000-07:002008-10-19T20:16:13.613-07:00A New Code Metric : Destroyed Lines Of Code (DLOC)We've all heard that counting lines of code, or <a href="http://en.wikipedia.org/wiki/Source_lines_of_code#Disadvantages">SLOC</a>, is a terrible way to measure a developer's performance. <a href="http://en.wikipedia.org/wiki/Function_points#Criticisms_of_Function_Points">Function point counting</a> is not a much better metric, just slightly different than SLOC. Instead, conventional wisdom says that good developers write fewer lines that get the job done better, so SLOC and function points do not reward that good behavior. (Although I admit SLOC does have its place in comparing two systems written by similar teams of developers)<br /><br />I propose a new metric that rewards good coding practice and simple, brief style : <strong>DLOC</strong>: <strong>Destroyed Lines of Code</strong>. You measure the bad lines of code that you remove from the system. If you can destroy lines of code, and the system is still working, then those lines of code were either bad, or just misleading bloat. You can also destroy lines of code by using packages that solve your problems - like destroying JDBC calls and replacing them with Hibernate mappings, or destroying your Factory Patterns and Service Locators and replacing them with Spring dependency injection.<br /><br /><p><strong>DLOC in Action</strong></p> <p>To illustrate DLOC, I worked on a system where I was asked to add some features. This system had an admin web interface with a few database tables. So I looked at the admin interface system - about 3000 lines of code, plus JSPs and 10 database tables. First run, I added a table, then changed all the old JDBC calls to Hibernate (which went faster than I expected). I ended up deleting some JDBC code so there were some destroyed lines of code right there. Next, I started really asking users about the app only to find out that nobody really needed the admin web interface. In fact, they really didn't need to edit the data that often at all. They would be perfectly happy with deploying changes at each release instead of making admins use a web interface. So I proposed removing the database tables and going to an XML config file with a similar schema, then removing the web interface altogether. The users could edit the XML and deploy it with each release when changes were needed. In the end, I destroyed all the tables, all the JSPs and much of the Java code, saving only the POJOs. I added some XML parsing, and in the end the whole system was only about 300 SLOC.</p> <p>If you analyze my performance by SLOC, I scored a terrible score - negative 2700. But by DLOC, I destroyed 2700 lines of code, and the users still got what they wanted. Hidden benefit: new data and features were simple to add to the tiny and simple system.</p> <p><strong>The rules: DLOC Methodology</strong></p> <p>Any metric needs rules so that performance can be measured fairly, so DLOC needs some rules:</p> <ul><li>count the lines of code that were destroyed (removed) from the system, bigger numbers are better (in contrast to golf scores where lower numbers are better)</li><li>destroyed comments also count - comments can lie, and removing bad comments is value added to your system</li><li>lines of code you added while changing the system do not count for you nor against you, after all, we don't know if those new lines are any good. We already know that we dislike SLOC, so we'll avoid looking at added lines and only count destroyed lines</li></ul> <p>Obviously this DLOC system <a href="http://www.johnhasson.com/2006/05/microsoft-has-pointy-haired-boss-named.html">can be cheated</a> just as easily as SLOC or function points: I could write a few thousand lines, only to purposely destroy them later. Also, you would expect a low DLOC for a brand new system where you've got to start from scratch (it seems I rarely have this luxury). DLOC is truly interesting on a system that is aging and even more interesting if the system was not constructed quite so perfectly.</p> <p><strong>Proper DLOC Usage </strong></p> <p>Truly caring for a software system has caused me to apply different techniques to make the software better. I try to use <a href="http://en.wikipedia.org/wiki/Agile_development">Agile Development</a>, <a href="http://en.wikipedia.org/wiki/Test_driven_development">Test Driven</a>, and even <a href="http://en.wikipedia.org/wiki/Broken_windows">Broken Windows Theory</a> when I am working on a system. DLOC is more of an observation than a methodology, but I think it's rewarding and fun to measure DLOC while improving a system. So here are some ways you can make your software system better, and increase your DLOC score too:</p> <ul><li><a href="http://www.harpoontech.com/blog/index.php/2007/10/17/stealth-mode-upgrade-your-system-and-dont-tell-everyone/">Introduce new technologies</a> into your system: Hibernate, Spring, even converting from Java 1.4 to Java 5 can reduce complexity and destroy lines of code (e.g. annotations, typed collections)<br /></li><li>Ask your customers what they like/dislike about the system, you may find that certain parts of the system can be removed or refactored</li><li>When you see code that could be improved / destroyed, change them immediately, don't procrastinate, this is Broken Windows for software</li></ul> <p>So improving software is the goal. And DLOC is a fun metric you can use to measure the changes you make. You can use DLOC to prove to yourself that the changes you made were a big impact on the system. Be proud of yourself and please help that poor software become better by destroying those unneeded lines of code. If you don't destroy them, who will?</p><p>-Jay Meyer</p><p>jmeyer at harpoontech dot com<br /></p>Jay Meyerhttp://www.blogger.com/profile/02158043537042803017noreply@blogger.com8tag:blogger.com,1999:blog-7030997386028100204.post-36074398584012408872008-05-16T09:54:00.000-07:002008-10-19T16:04:51.564-07:00Cluster your application - NOW!<div xmlns="http://www.w3.org/1999/xhtml"><p>Is your Java Application Server clustered? Why not? It should be, and your reasons for <strong>NOT</strong> clustering have vanished, so there's no excuse. Cluster. Now.</p><p>When designing a web architecture for an application, this question always comes up: "To cluster or not to cluster?" Clustering allows two or more app servers to act as one. This has many advantages, such as high availability if one server goes down, or the cluster can enable seamless upgrades to software or hardware. Clustering also allows a successful application to scale by spreading users across more and more servers. Unfortunately, clustered servers are much too rare today. But they shouldn't be rare, clustering should be the default, the normal case.</p><p><strong>Question: To cluster or not to cluster?</strong><br />At first the answer is "no", I mean... clustering make the installation more complex, there are performance impacts, and configuration problems and... well... <strong>RISK</strong>! But the benefits can be pretty great for an app that is growing. So if we can eliminate the <strong>RISK</strong>, then we have no reason <strong>NOT</strong> to cluster.</p><p><strong>Answer: Always Cluster.</strong><br />The time for clustering every Java application has come. Worrying about statefullness was a habit we acquired when SFSBs in EJB2 were dangerously bad at clustering (and a bad solution for many applications). But today, JBoss, Tomcat, and other Java app servers have matured to make clustering a solid part of any server. <a href="http://www.terracotta.org/" title="Terracotta">Terracotta</a> has even come in to save the day for clusters with large node counts, or with huge memory needs, so the reasons <strong>not</strong> to cluster become smaller.</p><p>RedHat is <a href="http://www.redhat.com/rhel/virtualization/" title="virtualization">telling everyone to virtualize Linux</a>.. ALWAYS! And it makes sense. Servers out there need the flexibility of virtualization, and the reasons to <strong>NOT</strong> virtualize have vanished. It's time has come. Technologies like Xen and VMWare have improved, but more than the technologies, the people have matured to understand the benefits and risks of virtualization. So now it's packaging and marketing that are driving virtualization as the default answer.</p><p>Java application server clustering should now follow and become the default installation. The technologies are solid, so the clustering idea is just in need of <strong>good packaging</strong> such as easier installation and configuration utilities, and <strong>good marketing</strong> in the form of endorsements and documentation about the many benefits of clustering.</p><p>The benefits:</p><ul> <li><strong>High Availability</strong> - Server crashes, app stays up, 'nuf said, this bene' is obvious.</li> <li><strong>Scalability</strong> - more users? add more servers, clustering lets your app grow - also an obvious bene'</li> <li><strong>Hot Deployment </strong>- this bene' is rarely understood, but it will save your <strong>weekend</strong>! Stop deploying on a weekend at 2am! Deploy on Tuesday at 11am, yes, <strong>while users are still using the system</strong>.... You see, most code changes do not require huge data conversions and outages (although those cases do happen). Most deployments are to fix a bug, change a label, add a brand new feature, or add more memory to the server - and those changes do not require downtime if I have a cluster and a deployment process that can make it <strong>HOT</strong>. Its quite simple - 1)take a server node out of the cluster, 2) upgrade the code, 3) put it back in the cluster, 4)repeat on other nodes. No late nights, no weekends -and no lost sleep. Developers and admins are handy and awake if anything goes wrong, no need to start that late-night conference call. Freedom...</li></ul><p>Believe it? You may not, but I've witnessed it. My system took a millions of hits a day, and we deployed twice a day at times - between 8 and 5. We had a cluster of JBoss application servers and did the deployments in broad daylight without any affect on users. And all this with out-of-the-box clustering from JBoss.</p><p>If you want to talk <strong>RISK </strong>- try taking an outage at 2am on a Sunday for 5 hours after 3 months of intense development by a large team. Put in a massive code change and pray that the deployment goes well... Now that's the stuff that nightmares are made of.</p><p>So cluster your Java servers, the benefits are big, and clustering comes with every application server - it probably came with the server you've got. No excuses, do it now.</p><p>-Jay Meyer</p></div>Rajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.com0tag:blogger.com,1999:blog-7030997386028100204.post-91639706802433028732007-10-17T09:52:00.000-07:002008-10-19T16:05:33.883-07:00Stealth mode: upgrade your system and don't tell everyone<div xmlns="http://www.w3.org/1999/xhtml"><p>Want to try JPA or Spring? But you have an old software system, and don't have the luxury of a clean new software project?</p><p><strong>Don't wait!</strong><br />Add Spring and/or Hibernate to your old Java software system. Instead of waiting, put in a STEALTH upgrade that is low-friction, low cost, and low visibility to the customers and others around the project. You can boast later when the system succeeds because of the new technologies.</p><p>We've all made some excuses for NOT upgrading the technology of a system. (But we can address each of these):<br />* I can't afford to overhaul the whole system<br />* Our Java servers don't support it, we cannot upgrade the Java server<br />* What will my manager or customer think?</p><p><strong>Upgrade the whole system?</strong><br />No software engineer has time & money to do that. So I suggest looking for a new feature and only upgrading that part of the system: this is the low-friction part. There's no need to overhaul every working DAO, leave them for later until you need to change them to keep costs down. Maybe you've got a new model object you are adding, why not use Spring and Hibernate to manage that object? Who needs to drag around the old EJB2 patterns, or your own home-grown JDBC DAO's just to keep the system consistent? In the end, you may finish the feature upgrade faster because you used Hibernate and Spring Transactions. After you try it, you may decide to convert other DAO's as changes happen to those parts of the system. This can all be done under the radar - Stealth mode - without kicking off a major overhaul of every DAO in the system.</p><p><strong>Upgrade my app server?</strong><br />You may think that your Java App server doesn't support Hibernate or Spring, but Hibernate and Spring are simply jar libraries that can be deployed inside your web app, and do not require an upgrade to your server. So go ahead and add the jar files inside your web app (or WAR or EAR file), and use them freely. If you'd like to use EJB3, and you have an app server that does not support it like Tomcat or IBM, you could use <a href="http://labs.jboss.com/jbossejb3/">JBoss Embeddable EJB3 container</a><a href="http://labs.jboss.com/jbossejb3/%29-">-</a> yet another free, OSS jar library that can be put inside your web app. Hibernate and Spring can even work in Java 1.4, (EJB3 and JPA need Java5).</p><p><strong>What will people think?</strong><br />Your customers and users of the system may not be able to spell JDBC, but that's OK, besides that is why they pay YOU, the software engineer. So they will not care whether you use Hibernate, OpenJPA, iBATIS, or raw JDBC. Many of your managers will not care either, as long as it doesn't cause the boat to rock in server operations (and it won't as described above in upgrading your app server). If your customer/manager is curious, you can tell him that many good things will be coming: like short development times, developers will be excited about the new technology. If you need to convince other developers, remind them that these technologies are popular for a reason, and its a good skill for a software developer to have for the future.</p><p>Marketing this kind of strategy is a challenge. Selling this idea to your customers, colleagues, and managers is probably a bigger challenge than actually making the software changes. Just keep in mind that some people will not care, and don't need to be scared by change, so leave them out of the conversation except to say that their new features will be coming soon. NOT telling them is not deception or evil, its just too much information (TMI), and it can sometimes be scary. So if they keep asking questions, tell them everything, but remember to tell them that confidence is high, risk is low, and everyone is doing it. The software people who know the system should get excited, not scared, by the new technologies adding power and speed to the system. Consider the alternative - try to hire a Java developer by telling him he gets to maintain old EJB2 CMP. Software engineers run away from those technologies, you won't get anyone to hire on without lying, and your current developers will leave in short order too.</p><p>The Stealth approach can avoid the hoopla and resistance that happens when any kind of change is suggested. So take advantage of the trust they place in you to make the right technology choice. You will get to use new technologies, your system will be better, and nobody needs to be the wiser. In the end you may get it all done faster and easier too.</p><p>-Jay Meyer</p></div>Rajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.com0tag:blogger.com,1999:blog-7030997386028100204.post-21754036180998713152007-10-05T15:53:00.000-07:002008-10-19T15:54:01.424-07:00Harpoon Technologies Announces Terracotta PartnershipHarpoon Technologies and Terracotta Partner to Provide Consulting Services<em> October 5, 2007</em><br /><br />Harpoon Technologies Open Source Experts to Provide Consulting<br /><br /><strong>ST. LOUIS – (October 5, 2007)</strong> – Harpoon Technologies, provider of open source consulting services, and Terracotta, a leader in infrastructure software for enterprise Java high availability and scalability, today announced a partnership to respond to growing demand for help implementing Terracotta clustering<br /><br />Terracotta offers IT organizations a lightweight approach to scalability that lowers costs and simplifies deployment by reducing development effort and easing the load on application servers and databases. Terracotta uses high-performance mapping of server memory changes, called Network-Attached Memory, to share temporary “work-in-progress” data among servers. That makes an application highly available without placing such temporary data in an expensive relational database. It also provides dramatic cost savings and much higher performance and scalability than either databases or application-tier caches.<br /><br /><strong>About Harpoon Technologies<br /></strong>Harpoon Technologies is a provider of Open Source Consulting Services. Harpoon Technologies offers performance tuning, custom development, and training. The company is headquartered in St. Louis. For more information, please visit <a href="http://harpoontech.com/">www.harpoontech.com.</a><br /><br /><strong>About Terracotta, Inc.<br /></strong>Terracotta’s infrastructure software provides affordable and scalable high availability for Java applications. Companies use Terracotta to offload work from databases and application servers and to reduce their development efforts. Founded in 2003, Terracotta, Inc., is a private firm headquartered in San Francisco. More information is available at www.terracottatech.com. Terracotta’s open source community is available at <a href="http://www.terracotta.org/">www.terracotta.org</a>.Rajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.com0tag:blogger.com,1999:blog-7030997386028100204.post-8180789885640855292007-08-10T15:55:00.000-07:002008-10-19T15:56:11.848-07:00Writing Secure ApplicationsThis presentation was originally given at the <a title="STL JUG Presentation" href="http://www.ociweb.com/javasig/knowledgebase/2007-08/index.html">St. Louis Java user Group</a>.<br /><br />Security topics in the internet age remain esoteric and the domain of experts. Firewalls and Intrusion Prevention Systems are only parts of the complete security picture. Application security is an essential piece of the security puzzle, and without it, the sensitive application is still in jeopardy, even with the strongest network and OS security. Yet many developers lack the knowledge needed to protect their sensitive data using secure application development techniques.<br /><br />The presentation focuses on tools and techniques that developers can use write a secure application from scratch, or test an already-installed application. Attacks will be discussed such as SQL injection, cross-site scripting, dictionary attacks. These attacks can be foiled using testing tools, Java Cryptography (JCE), and secure design techniques. Tools and code samples will demonstrate these techniques so that you may apply them to your applications. We'll also look at Risk Assessment, impact analysis and the bigger picture of security or SOx audits, in case your system is audited.<br /><br /><a title="WritingSecureApps.pdf" href="http://www.harpoontech.com/blog/wp-content/uploads/2007/08/WritingSecureApps1.pdf">Download the presentation here. </a><br /><br />-Jay MeyerRajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.com0tag:blogger.com,1999:blog-7030997386028100204.post-24311478956808105042007-08-09T09:44:00.000-07:002008-10-19T16:06:15.617-07:00How to avoid SQL injection in Hibernate (A Hibernate Urban Legend)<p>Somewhere along the line java developers came to believe that Hibernate protects you from SQL injection. I'm not sure where they came to believe that. Maybe it is because you no longer have to write SQL and Hibernate does many other magical things - it has to protect you against SQL injection.</p><p>I'm tired of telling java developers that HQL has the same vulnerabilities as SQL, they don't believe me and think Hibernate offers them some sort of magical protection from bad HQL. Basically, what I term bad HQL is when named parameters are not used. Consider the following example:<br /></p><blockquote><p>String goodParameter="Raj lane";<br /><br />Query badQuery = session.createQuery("from Address a where a.street='"+goodParameter+"'");</p></blockquote><p>I have SQL logging turned on so I can see that the generated SQL is as follows:</p><blockquote><p>select address0_.addressId as addressId, address0_.street as street1_ from Address address0_ where address0_.street='Raj lane'</p></blockquote><p>Now consider the following where I attempt "HQL Injection"</p><blockquote><p>String badParameter="la' or '1'='1";<br /><br />Query reallyBadQuery = session.createQuery("from Address a where a.street='"+badParameter+"'");</p></blockquote><p>And the resulting SQL:</p><blockquote><p>select address0_.addressId as addressId, address0_.street as street1_ from Address address0_ where address0_.street='la' or '1'='1'</p></blockquote><p>Note that the above SQL passes the parameter directly into the SQL. The generated SQL will return all rows in the table. Which is bad, but SQL injection opens us up to much worse attacks. So the moral of the story is to use named parameters, the above code can be fixed as follows:</p><blockquote><p>String badParameter="la' or '1'='1";<br /><br />Query reallyBadQuery = session.createQuery("from Address a where a.street=:street");<br /><br />reallyBadQuery.setParameter("street", badParameter);</p></blockquote><p><a href="mailto:rpatel@harpoontech.com">Rajesh Patel</a><br /><br />Harpoon Technologies</p>Rajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.com3tag:blogger.com,1999:blog-7030997386028100204.post-13531377814218352982007-07-19T15:56:00.000-07:002008-11-03T11:13:41.677-08:00Harpoon Technologies Expert PresentationsHarpoon Technologies employs many open source experts. We have shared our knowledge by giving presentations on EJB3, JSF, JBPM, XFire, Jabber and Seam. We strongly believe that these technologies represent the best of breed in the java space.<br /><br />All of our presentations can be located at <a href="http://www.harpoontech.com/services/presentations/">http://www.harpoontech.com/services/presentations/</a><br /><br />If you would like for us to give one of these presentations at your company, please let us know.<br />Rajesh Patel<br /><a href="emailto:rpatel@harpoontech.com">rpatel@harpoontech.com</a>Rajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.com0tag:blogger.com,1999:blog-7030997386028100204.post-67307227178391852802007-04-04T15:57:00.000-07:002008-10-19T15:57:48.168-07:00Seam PresentationQuite a large crowd turned out for the Seam presentation that I gave at the gateway JUG last night. The java industry appears to be in a pivotal moment in the shift from Struts. All the indications that I see is that Struts 2.0 is not going to take off, but JSF see significant adoption in the near future. Most new projects that I have heard about are choosing JSF!<br /><br />Here is my Seam presentation:<br /><a href="http://www.harpoontech.com/blog/wp-content/uploads/2007/04/SEAM.pdf"> Seam Presentation</a><br /><br />Raj<br /><br /><a href="mailto:rpatel@harpoontech.com">rpatel@harpoontech.com</a>Rajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.com0tag:blogger.com,1999:blog-7030997386028100204.post-72455774214239250172006-10-17T09:59:00.000-07:002008-10-19T16:04:11.633-07:00Where Did I Put My Encryption Keys?<div xmlns="http://www.w3.org/1999/xhtml"><p>As a software security consultant, I was asked to solve some interesting problems with cryptography in Java. I warn others that security is esoteric and an afterthought much of the time. So taking security into consideration early is better. But even if you are a late-comer, the solutions do not cost that much, especially when compared to losing valuable data to an unfriendly attacker. Most of them can be done with Java packages that come with the JDK (the .NET framework has them too).</p><p>For example, a super easy solution to a common problem is password hashing. Passwords need to be hidden from attackers, programmers, admins or DBA's. This can be easily done with the MessageDigest class in Java. The great thing about hashing is that there is no key to store, you simply run your password through the algorithm, like SHA, and out pops a byte array that you can store in the database or a text file. Then the next time you prompt for the password, you hash that password with the same MessageDigest function, compared it to the one you've got stored, and if they match, you are logged in. Attackers or admins cannot possibly guess what that hash is by looking at it. So there you have a 3-line maintenance-free security solution that will pass any security audit - it's the same technique most Unix systems use to store their passwords.</p><p>Software cryptography provides quite a few different solutions, and when you analyze them, you try to find the weak spot. The weak spot is usually the keys to a cipher like 3DES or AES. When you use a cipher, you have a key, and storing this key safely can be a big problem since you don't want them to be stolen or<br />compromised. Most security books point this out and then fall short of suggesting some solutions: e.g. <a href="http://www.amazon.com/Building-Microsoft-Applications-Pro-Developer-Paperback/dp/0735618909/sr=8-3/qid=1161092157/ref=sr_1_3/102-3192586-6603361?ie=UTF8&s=books">this one</a> from Microsoft Press tells you to put the key somewhere far away from the data, but the example code violates that very suggestion by putting the key and the data right next to each other in the Windows registry (with a disclaimer that says don't do this). So what hope does a novice software developer have to avoid making bad choices when storing the keys? Other books on secure software are no better, I think their publishers forbid them to give good examples for fear of being sued.</p><p>Bad practices happen, but I will suggest some better ones since I have no publisher to forbid it. Let's define some security requirements for cryptographic keys that every developer should follow:<br />* The keys should be hidden<br />* The keys should not be known to the programmer<br />* The keys should be changeable after installation in case the keys are lost to an attacker<br />That's all we need, really. Now lets try to meet these requirements.</p><p>Solution 1: Hard-coding the keys in the code. Many developers will create a static String in the code that has encryption keys in it. This gets compiled and shipped with the software. Since the customer normally gets binaries and not source code, this hides the keys pretty well to satisfy the requirement above. However, the next two requirements are impossible with this solution. Now the programmer knows the keys - imagine being kidnapped by spies and tortured for the keys, or more likely, imagine being bribed for the keys. Then imagine what happens if someone bribes a programmer and then posts the keys on a public web site. Even easier than bribery, an attacker might download the software from your <strong>own</strong> web site, decompile it (amazingly simple for Java or C#), and then post your keys to a web site. You must panic, update the code, and every one of your customers must upgrade ASAP. This is a nightmare. You might say, "Yes, but most people would not know how to decompile the code, right?" Remember, you are not protecting your keys from MOST people, you are protecting them from a motivated attacker, and motivated people <strong>DO</strong> know how to decompile your code. (Yes, of course I know how to do it, but I won't bother putting the details here.)</p><p>Solution 2: Putting the keys in a config file. Storing your keys in a config file seems unpleasant because the keys are right there in plain text for all to read. Clearly this violates the hidden requirement above, doesn't it? Its true - its not hidden very well. However, the config file <strong>DOES</strong> satisfy the other two requirements: the keys are hidden from the programmer and they are changeable later. So now when the keys are lost to an attacker who posts them on a web site, you tell the customer to change the config file to enter new keys. The other customers need not worry, and the customer who lost their keys will be secure again in seconds.</p><p>Well maybe we can encrypt the config file! Yes, then we solve the hidden requirement! Brilliant! Except for one thing - where do you store the key for <strong>THAT</strong> encryption? Hard coded in the app? In another config file? You can see where this is going... The truth about security solutions is that they can all be beaten, some solutions are simply better than others and cause the attacker to expend more effort. In the case of the keys in the config file, you can simply use Unix or Windows file permissions to limit the readability of the file to the people who need to know, thus limiting the attackers who can get the keys. Then if the keys end up on a hacker web site, you change the keys, then haul off the admins to jail and torture them - but at least the programmers are safe and your customers are safe.</p><p>Some companies actually have a security policy that says "no storing passwords in plain text on the disk". I would say that this rule is impossible to follow. Because when I encrypt the password, where would I store the keys for that? Its a rule made by people who haven't thought about it. Sometimes this rule leads to hard-coding the keys in the code, and that's even worse. But if you have to follow this silly rule, use <a href="http://en.wikipedia.org/wiki/Rot13">ROT13</a> since it needs no keys itself.</p><p>You can further protect the keys by putting them somewhere else separate from the data - maybe store them in a JNDI server instead of a config file (note that JNDI would persist to a text file, but at least it would be on a separate machine), or in a separate database from the rest of the data. (BTW: where did you store your database password? I suggest JNDI data sources, follow Solution 2 above). Anything to limit the exposure of the keys so you can hide them from most, but not all.</p><p>You can see that hard-coding the keys is appealing because it hides the keys, and what's more important than that? But hard coding the keys totally ignores the other two concerns, and it doesn't really do a great job of hiding the keys. So storing the keys in a config file (or other external place), even in plain text, is always preferred.</p><p>-Jay Meyer</p><p>jmeyer at harpoontech dot com</p></div>Rajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.com3tag:blogger.com,1999:blog-7030997386028100204.post-2426815817193833672006-07-13T15:58:00.000-07:002008-10-19T15:58:48.897-07:00Data Mining Research ProjectI've decided to start a new side project. Data Mining has been an intriguing subject that I've been interested in for some time now. I wrote an automated eBay parser a while back that would alert me to new auctions via IM every hour or so. That project was fun, but violated so many of eBay's terms of service that I no longer use it.<br /><br />I would like to come up with some kind of automated system for data retrieval and am in the initial phases of design/prototyping now. The design needs to be flexible enough to adapt to changing conditions encountered during the data gathering session, and also will need to be able to be controlled dynamically as my criteria change over time. Perhaps I'll integrate <a href="http://labs.jboss.com/portal/jbossrules">JBoss Rules</a> (formerly known as Drools) into this, as a means of applying rules to the data gathering process. I'll also integrate this with our XMPP service, so I can get updates on my phone as well as controll the process from anywhere.<br /><br />I'm not quite sure what I want the end product to do exactly, but I know what I don't want it to be: a Google replacement. I'm not looking to index the web, more like a "smart search" that is continuous and updates me in real time (or close to it anyway) as it discovers new data.<br /><br />Anyway, I hope to have something useful in the end. If you have any comments or suggestions, drop me an email.<br /><br />-Jason<br /><br />jwambach@harpoontech.comRajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.com0tag:blogger.com,1999:blog-7030997386028100204.post-68640173740624625432006-06-10T15:59:00.000-07:002008-10-19T15:59:35.909-07:00JSF Presentation: Seam leads the JSF packOn June 8, 2006, I gave the presentation about JSF to the <a href="http://www.ociweb.com/javasig/">St. Louis Java User Group</a>. I got a good reiview from <a href="http://www.weiqigao.com/blog/">one of the attendees</a>, and in general, I think everyone enjoyed the presentation. I agreed to do the presentation without knowing anything about the topic, but I read a book, looked at the buzz, and worked through two tutorial code examples myself to get comfortable: <a href="http://www.harpoontech.com/blog/wp-content/uploads/2006/06/JSF.ppt">JSF.ppt presentation file</a><br /><br />My opinion, Seam from Gavin King and JBoss has some really good ideas. And the crowd at the JUG seemed to agree. We had a few JSF users in the group that were using MyFaces - they were really impressed with it vs. Struts 1.x. The biggest lessons learned are that the surrounding technologies for JSF are still taking shape, but they are a feasible alternative to Struts. When you combine Facelets, Tomahawk, and Myfaces you get a powerful MVC replacement for Struts. And if you have Java5 and EJB3, Seam will take JSF right down those last miles of the n-tier architecture with suprisingly little effort.<br />The next project I start will be using Seam. The JSF and EJB3 technologies are the skills that I need for the future of Java Server development.Rajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.com0tag:blogger.com,1999:blog-7030997386028100204.post-10055636216142326992006-06-06T15:59:00.000-07:002008-10-19T16:00:09.905-07:007-Layer DipJava architects are insecure. They make the big decisions and try to make as many software layers as possible to maximize complexity. Somehow when entrusted with designing reusable software that is to be used by tens of Java programmers, software architects feel the need to make lots and lots of decisions, in order to confuse the poor souls who really just want to solve the problem and go home for a quiet evening of beer and Tivo. That's job security, it makes the Java architect indispensible to the project, calming their insecurities. It makes for large system that have layers and layer of code.<br /><br />I call this 7-layer-dip architecture. Its is an n-tier idea gone terribly wrong, putting layer after layer of architectural bloat on the code until its nearly impossible to figure out where to make changes to add something like a new field on my object & web page. Its like a 7-layer dip, lots layers and time consuming process to create a tasty appetizer.<br /><br />In a 7-layer dip - its all about the beans. If you take away the beans, you have a 6-layer dip, right? No, you do not. Instead you have some lame-o salad. Without the beans, you have something that's no longer Mexican, you have to dump out your margarita! The Fiesta is OVER! The other 6 layers are important, for without them its just beans and who wants that, its no longer fun without the other layers. Do I need 7? No. I think 4 layers is still pretty tasty, but we're enamored with the number 7, so we make up a few more.<br /><br />Analgoy-stretching time: In Java apps its also about the beans- the Java beans, the POJOs, the model classes - the simplest form of code, the useful stuff. Without the beans you have no application, just empty framework stuff, no fun, no margarita. But then some Java architects like to wrap these beans in 7 layers of tomatoes and lettuce and cheese and sour cream and tortillas because they think its necessary to keep the party going. This is where the party goes wrong - Java beans do not need 6 more layers of wrappers and EJBs and services and business delegates and data accessors and views and actions. These layers do not add to the fun of writing software. Instead, they serve to alienate developers from the important parts - the beans, the app, ...the user. Developers then try to make themselves look smart by jumping on the buzzword bandwagon. I should never hear a developer utter a phrase like this in front of a user, "well, I can just make the action return a new implementation from the business delegate factory". Perhaps Java developers boast of bloat-buzzwords so they can raise their pay, but if the users are smart, they will just be annoyed and they'll hire that PHP script kiddie from the basement next door.<br /><br />In the late 90s we all let Sun talk us into n-tier machines with Entity Beans, CORBA, and SNA spread out in 5 times zones, so we could write "Enterprise applications!" (and buy some E-class Sun hardware). We took in some good dot-com paychecks and the accountants and managers rode our e-coat-tails to high stock prices. Good for us! In the meantime, 8th graders were selling T-shirts with PHP apps running on old hardware in their basement. And they did it all that by reading "Learn PHP in 7 minutes". Sure, that little PHP app is not ready to handle 8000 hits per minute, but its built to be the right size for selling T-shirts out of a basement, and when he needs to add the baby-tee sizes and the 3XL sizes to the web site, he can do it in a day by recoding some PHP and adding some columns to his MySQL table.<br /><br />Don't panic, I'm not advocating the destruction of design patterns, I still advocate the use of Factory and MVC and lots of other good designs. But they are simply being overdone. For example, I like to use the Factory pattern if I'm pretty sure I have two different impls to hand out in different situations. A SecurityService is a good example for just about any app, because I have changing needs from development to production. I can write one SecurityService for LDAP and one for reading a password file to authenticate users. I use a SecurityFactory to return the correct service, so I can change it out for test vs. production by changing a simple config file. Great. There, I used a factory pattern in a good way to swap out two different pieces of code seamlessly. But I don't need a Factory for every bean in my system. When am I really going to need two different impls for a Book object? I mean if the web site sells Books, there's probably only one way to sell books, do I need a Factory to pass out two different classes of Book? Almost never. But this kind of knee-jerk over-architecture decision is everywhere even in closed-source corp IT systems. If, by chance, I invent another impl for a book, I can slam a BookFactory in later while I'm putting in the new BookImpl - easy stuff (even easier, use Spring). But making a BookFactory in an application is usually just obnoxious. The only reason to do that is to pad your own resume with bloatware knowledge. I know..I know.. you can contrive the need for a PoolingBookFactory and a SimpleBookFactory and a WebServiceEnabledBookFactory. But most of the time you just need a Book - new Book(), its just that simple. Very few cases arise where I need the kind of flexibility a Factory pattern provides, but develoeprs and architects don't even remember the reason for the complex case and they abuse that pattern for a much simpler case like Books just in case they'll need it later.<br /><br />This kind of over-architecture slows down developers, especially new ones who must learn this annoying bloat-itecture before they can add a new simple property to the Book class like Book.setOprahPickedIt(boolean b).<br /><br />Why does this happen? What makes this bloat so prevalent? Who promotes these guys to Architect positions and lets them run big software projects? It must all come from the old waterfall addage that big code now is cheap, and change later is costly, RUP uses this principle to justify their brand of $10K tools. That theory then pushes the derivative theory: "why overarchitect later what you could over-architect today?" So IT managers believe a Java architect when he hears "well we need all this code to scale and to be prepared for change", and the checkbook writes a big check for "architecture and design" with zero expectation of a demo or milestone release coming anytime soon. And admittedly, I've jumped onto a project and I drank the kool-aid and gave the architect the benefit of the doubt "certainly there must be a really good reason for all this code...". And I changed piles of code to add a single property to a class. But when I turn to ask a peer whose been grueling with the system for 10 months, that poor coder is hard-pressed to answer the question "why?". This doesn't just happen on old EJB projects, it happens on EJB-less Web projects and thick-client apps too. Reminds me of this old parable: <a href="http://wicksie.com/articles-published-by-tiggeronspeed/think-outside-the-box/">why we do things</a>.<br />The best things to cure my pain are the Agile warriors and the POJO bigots. Both of these groups have shown me that simple is better and that change is pretty cheap. If you build your code in a simple way first, then strap on some good features one by one carefully, your code doesn't need to blow up into a confusing mess and you'll be agile when the customers change their minds or ask the system to do more. I like the Spring and EJB3 movements for reminding us that simple is good. And I like the Agile movement for the idea that change is inevitable, trust your customers and don't hire a lawyer to write that contract. Now we just need to convince all the decision makers that this is the best way.<br /><br />Little piece of advice to those architects of bloat. Embrace change, keep thing simple, and read some software articles written in this decade. I recommend <a href="http://www.theserverside.com/">theServerSide</a> and the <a href="http://agilemanifesto.org/principles.html">Agile Manifesto</a> .<br /><br />Jay Meyer<br /><br />jmeyer@harpoontech.comRajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.com0tag:blogger.com,1999:blog-7030997386028100204.post-73780404423132595112006-06-05T16:00:00.000-07:002008-10-19T16:00:50.361-07:00My Experiences with JSF<style media="screen,projection" type="text/css"> pre { padding: 1em; border: 1px dashed #2f6fab; color: black; background-color: #f9f9f9; line-height: 1.1em; </style><br /><br />I am currently converting a large Struts application to JSF. My first instinct was to simply replace the Struts tags with the equivalent JSF tags and JSTL.<br /><br />Turns out things aren't so simple. JSF & JSP don't really play all that well together as detailed in the following article: <a href="http://www.onjava.com/pub/a/onjava/2004/06/09/jsf.html">Improving JSF by Dumping JSP. </a> So this means if you mix normal HTML and JSP tags you may run into unexpected problems. To avoid this you must fully understand the rendering lifecyle of JSPs and JSF. Talk about a maintainability nightmare. If you want to avoid this, you must buy into converting your old JSPs tag for tag to the equivalent JSF tags. That doesn't seem easier to me. But wait, the tools are here to save us. They didn't save us from the nightmare of EJB 2.1, they won't save us now.<br /><br />The framework <a href="http://www.jsfcentral.com/articles/facelets_1.html">Facelets</a> was created to solve this issue by by completely replacing the JSP layer we all know and hate. It drops the need to fully convert your app over to JSF tags, you simply use HTML plus JSF tags where needed. Great, I found it frustrating to use the JSF markup to output the HTML I wanted to display in the first place. No more! Also, no more crappy code generation on the backend. Todays' date isn't 1998, lets move on.<br /><br />Facelets make JSF tolerable but still I think it should be more simple. Why does JSF introduce its own dependency injection when EJB3 already has one? Because JSF came out before EJB3! Enter <a href="http://www.jboss.com/products/seam">Seam</a> the latest from Gavin King(author of hibernate) which vastly simplifies the JSF configuration using annotations and tightly integrates JSF with EJB3 and JBPM<br />The <a href="http://blog.hibernate.org/cgi-bin/blosxom.cgi/2006/05/16"> Web Beans JSR</a> aims to standardize Seam.<br /><br />My prediction is that Web Beans technology will be the BFG that allows them to punch holes in IBMs' armour and finally makes web application implementation simpler.<br /><br />Rajesh Patel<br /><a href="mailto:rpatel@harpoontech.com">rpatel@harpoontech.com</a><br />Harpoon TechnologiesRajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.com0tag:blogger.com,1999:blog-7030997386028100204.post-30755050554101580892006-06-04T16:01:00.000-07:002008-10-19T16:03:21.442-07:00Webservices and HardwareI have posted the slides and the source code for the xfire integration demo that I presented on 4/5/06. For those of you who were not in attendance, this demonstration's purpose was to introduce XFire and how it can be used to expose a web service. Rather than the typical "Hello World" example, I decided to integrate the service with a microcontroller, whose purpose is to display the temperature of my basement.<br /><br />All of the source code and slides can be located here:<br /><a title="XFire client source" href="http://testwww.harpoontech.com/presentations/TempClient.zip">Client</a> (12.8M)<br /><a title="XFire web service source" href="http://testwww.harpoontech.com/presentations/TempService.zip">Service</a> (4.1M)<br /><a title="XFire demo slides" href="http://testwww.harpoontech.com/presentations/xfire.ppt">PPT Slides</a> (308K)<br /><br />I have not included all of the rxtx libraries (the .dlls) as their installation is platform specific. The hardware schematic is included in the ppt slideshow, if you need a detailed parts list, just ask me.<br /><br />-<a href="mailto:jwambach@harpoontech.com">Jason Wambach</a>Rajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.com0tag:blogger.com,1999:blog-7030997386028100204.post-82757854592258137582006-03-27T16:01:00.000-08:002008-10-19T16:02:42.093-07:00EJB3 PresentationI recently gave a presentation on EJB 3.0 at my<br />local javasig. They were an astute crowd<br />and pretty much asked me everything that<br />I chosen not to cover.<br /><br />However they did not ask me why they should use<br />EJB3 over spring + hibernate. The main benefit<br />of EJB3 is that it will be standard based. But<br />functionally spring + hibernate has it beat.<br /><br />Here is my presentation for anyone who is interested:<br /><a href="http://rajix.com/EJB3_stljavasig.pdf">EJB3 Presentation</a><br /><br />Raj Patel<br /><br />rpatel@harpoontech.comRajesh Patelhttp://www.blogger.com/profile/11070763300226168187noreply@blogger.com1