Bulk loading data in to databases

Submitted by Falken on

There was a thread on CF-Talk about how best to split a large file up into records and then import them into a database - it happened to be MySQL but this note applies to every database I've ever used (including Oracle and MSSQL).
The quick summary is 'don't use ColdFusion for this'. Sometimes all you have is a hammer, and so everything looks like a nail - but there are much much more robust and much much faster ways of doing this.

The specifics vary for each database engine, but in MySQL, for instance all you need to something like

<CFQUERY datasource="mysql" .......
load data infile 'e:/inetpub/webroot/real_estate_data/....dx_custom/format1.txt'  
into table hmls_residential_temp  
lines terminated by '\r\n'  
ignore 1 lines
</CFQUERY>

LOAD DATA INFILE can load tens of thousands of lines of data in a very short period of time, from a file local to the ColdFusion server (or the database server).

So, next time you see a nail, check if you really need a hammer :-)

Sections