Jump to content
joandcruz

FM and web scraping

Recommended Posts

joandcruz

I am trying to work on a personal database that would scrape a certain website and put that information into a database

 

I have been working on the excel / access route but on looking at the features of FM8.5 it seems that you can do the job totally in the FM program

 

Can anybody give me any info on setting up an automated process with FM to scrape certain parts of a page on a website based on a common link and viewing the results in a table. Is this a pretty standard feature of any database program or is FM particularly suited to this?

 

As you can probably tell this is by no means my area of expertise but I have the free trial and I am ready to go

Share this post


Link to post
Share on other sites
AHunter3

a) Create a webviewer on a layout and give it a name. In the example below I am naming my web viewer "WebScraper". Don't omit the naming step. It will not work if you do not name your web viewer!

 

b) Either natively (as part of the webviewer definition) or via scrip step "Set Web Viewer", load the page you intend to scrape in the web viewer. Give it time enough to load (i.e., Pause for a relevant # of seconds)

 

c) Set a $Variable to GetLayoutObjectAttribute ( "WebScraper" ; "content" ) where "WebScraper" is the name you gave to your web viewer

 

d) Examine raw code from the web site itself. Most web browsers will show you the source code. Using text parsing commands such as Middle and Position and Length and PatternCount and so forth, isolate the relevant chunk or chunks of text from the $Variable and make use of them as you see fit — make new records or populate fields in existing records, etc

 

 

Here is an example using http://themoneyconverter.com/GBP/EUR.aspx to snag the current conversion rate between British pound sterling and euro:

 

Go to Layout ["Layout with Web Viewer"]

Pause [ 1 second]

Set Variablel [$Code; GetLayoutObjectAttribute ( "WebScraper" ; "content" ) ]

Set Variable [$Snippet; Middle ($Code; Position ($Code; "Latest Exchange Rates: 1 British Pound Sterling ="; 1; 1); Length ($Code) )]

Set Variable [$Snippet2; Middle ($Snippet; Position ($Snippet; "= "; 1; 1)+2; Length ($Snippet) ) ]

Set Variable [$Pound to Euro; Middle ($Snippet2; 1; Position ($Snippet2; " Euro"; 1; 1)-1) ]

Edited by AHunter3

Share this post


Link to post
Share on other sites
This thread is quite old. Please start a new thread rather than reviving this one.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.




×
×
  • Create New...

Important Information

Terms of Use