#193552 - 2009-04-22 11:30 PM
Any way of speeding up an import from Excel to Access...
|
Viggen
Starting to like KiXtart
Registered: 2002-03-22
Posts: 110
|
Hi all!
Got a little speed issue here... 
When I run the following script it takes 14.5 minutes to import 18200 lines from a csv file to an Access DB.
If I do the Import from Access it takes less than 5 sec. to do all 18200 lines 
break on
$nul = setoption('WrapAtEOL','On')
?? "[1] : " + @time
$K2E_EXEPath = @scriptdir
$lokaldb = $K2E_EXEPath + "\Artiklar.mdb"
$importfile = loadfile(@scriptdir + '\090420.csv')
$i_array = quicksort($importfile,0)
?? "[2] : " + @time
$ConnDSN = "Driver={Microsoft Access Driver (*.mdb)};Dbq=" + $lokaldb + ";Uid=;Pwd=;"
$sql = "CREATE TABLE tmptbl (Artikelnummer varchar (255), Benämning varchar (255), Styckpris double, Flag char (1), EAN varchar (255))"
$rs = DBCommand($ConnDSN,$sql)
?? "[3] : " + @time
For each $row in $i_array
If Not $row = ''
$items = Split($row,";")
If not Instr($items[0],'Artikel')
$aitem = Cstr($items[0])
$bitem = Cstr($items[1])
If Instr($bitem,"'")
$bitem = Join(Split($bitem,"'"),"")
Endif
$citem = Join(Split($items[2],' '),'')
$ditem = Cstr($items[3])
$values = "'" + $aitem + "','" + $bitem + "','" + $citem + "','" + $ditem + "'"
$fields = "Artikelnummer,Benämning,Styckpris,EAN"
$sql = "INSERT INTO tmptbl(" + $fields + ") VALUES(" + $values + ")"
$rs = DBCommand($ConnDSN,$sql)
$items = ''
Endif
Endif
Next
?? "[4] : " + @time
Get $x
I have looked at the example here, that looks like this: (would be nice to get the info from the xls, instead of saving it to a csv first)
Dim cn As ADODB.Connection
Dim strSQL As String
Dim lngRecsAff As Long
Set cn = New ADODB.Connection
cn.Open "Provider=Microsoft.Jet.OLEDB.4.0;" & _
"Data Source=C:\test\xltestt.xls;" & _
"Extended Properties=Excel 8.0"
'Import by using Jet Provider.
strSQL = "SELECT * INTO [odbc;Driver={SQL Server};" & _
"Server=<server>;Database=<database>;" & _
"UID=<user>;PWD=<password>].XLImport9 " & _
"FROM [Customers$]"
Debug.Print strSQL
cn.Execute strSQL, lngRecsAff, adExecuteNoRecords
Debug.Print "Records affected: " & lngRecsAff
cn.Close
Set cn = Nothing
But I can't figure out how to get the strSQL-line to work with DBCommand()
Or is there another way to get closer to the 5 sec. result that I get in Access?
/Viggen
|
|
Top
|
|
|
|
#193554 - 2009-04-23 01:43 AM
Re: Any way of speeding up an import from Excel to Access...
[Re: Viggen]
|
Howard Bullock
KiX Supporter
   
Registered: 2000-09-15
Posts: 5809
Loc: Harrisburg, PA USA
|
Which section of your code is slow? You have 4 outputs for time, please post the output results so we can see how long each section takes.
|
|
Top
|
|
|
|
#193555 - 2009-04-23 01:54 AM
Re: Any way of speeding up an import from Excel to Access...
[Re: Howard Bullock]
|
Howard Bullock
KiX Supporter
   
Registered: 2000-09-15
Posts: 5809
Loc: Harrisburg, PA USA
|
$importfile = loadfile(@scriptdir + '\090420.csv') $i_array = quicksort($importfile,0)
The loadfile UDF I found can load a file into an array ir directly into a variable. You did not post your UDF code so I can not determine if you intended read the file into a variable or into an array. However, the quicksort UDF is used as if you expected that $importfile was an array.
Function loadfile($file, optional $array, $Uni)
DIM $fso,$f,$fs
if $uni $uni = $uni-3 else $uni = -2 endif
if not $uni $uni = -2 endif
$fso = CreateObject("Scripting.FileSystemObject")
$f = $fso.GetFile($file)
If @ERROR Exit 2 EndIf
$fs = $f.OpenAsTextStream(1,$uni)
if not $array
$loadfile = $fs.Read($f.size)
else
$loadfile = Split($fs.Read($f.size),$array)
endif
Exit @ERROR
EndFunction
function quicksort($a, optional $indexcol )
;$a array to be sorted
;$indexcol index of subitem to sort on
|
|
Top
|
|
|
|
#193556 - 2009-04-23 03:04 AM
Re: Any way of speeding up an import from Excel to Access...
[Re: Howard Bullock]
|
Viggen
Starting to like KiXtart
Registered: 2002-03-22
Posts: 110
|
Ah! my bad. sorry
It is the "For each $row... Next"-part that is slow.
The Loadfile that I use looks like this:
Function loadfile($file)
DIM $fso,$f,$fs
$fso = CreateObject("Scripting.FileSystemObject")
$f = $fso.GetFile($file)
If @ERROR
Exit(2)
EndIf
$fs = $f.OpenAsTextStream(1)
$loadfile = Split($fs.Read($f.size),@CRLF)
Exit(@ERROR)
EndFunction
The output looks something like this:
[1] : 02:31:04 [2] : 02:31:11 [3] : 02:31:11 [4] : 02:43:08
[3] to [4] takes 12 mins. (I have removed some data from the csv)
The thing is that in the same time as this script go from [1] to [3], Access has imported everything.
That's why I wonder if the "SELECT * INTO"-approach would be faster.
/Viggen
|
|
Top
|
|
|
|
#193561 - 2009-04-23 12:48 PM
Re: Any way of speeding up an import from Excel to Access...
[Re: BradV]
|
Viggen
Starting to like KiXtart
Registered: 2002-03-22
Posts: 110
|
That sounds like a plan, but...
I have no clue on how to do that 
/Viggen
|
|
Top
|
|
|
|
#193563 - 2009-04-23 01:49 PM
Re: Any way of speeding up an import from Excel to Access...
[Re: Viggen]
|
Howard Bullock
KiX Supporter
   
Registered: 2000-09-15
Posts: 5809
Loc: Harrisburg, PA USA
|
Using ODBC to insert a record at a time will be painful. The approach you nned to take is to leverage the programability of Access itself. You should be able to find ways to utilize Access' import wizard functionality via script. The result should be import speeds the same as interactive Access use.
I did not find any quick access to the importWizard although I did see some references.
Using ADO may be faster: How Can I Import a .CSV File into an Access Database?
|
|
Top
|
|
|
|
#193582 - 2009-04-24 07:44 AM
Re: Any way of speeding up an import from Excel to Access...
[Re: Howard Bullock]
|
Viggen
Starting to like KiXtart
Registered: 2002-03-22
Posts: 110
|
The reason for scripting this is that, the client-computer has no MS Office.
I followed that link Howard, and I can only say, OH... My... God!
I have changed the script and now the time output looks like this 
[1] : 07:35:25 [2] : 07:35:33 [3] : 07:35:33 [4] : 07:35:36
From 12 Mins. to 11 Sec.
Now that is a very good way to "speeding up an import..." 
Thank you very much!
/Viggen
|
|
Top
|
|
|
|
Moderator: Glenn Barnas, NTDOC, Arend_, Jochen, Radimus, Allen, ShaneEP, Ruud van Velsen, Mart
|
1 registered
(Allen)
and 675 anonymous users online.
|
|
|