Quantcast
Channel: Recent Questions - Stack Overflow
Viewing all articles
Browse latest Browse all 18095

PowerQuery M: import data from 10 websites and put it into a single table

$
0
0

Could someone please help me fix this VBA code so that the output doesn't have any null rows and all data for a particular date is displayed parallely? Also, it'd be great if you could optimize this macro (it takes sooo long for it to load) so that it doesn't download all data from those website but just accesses these two containers from each website to download tables:

  1. XPath="/html/body/main/div/div[4]/div/div/div/div[3]/div[1]/div/div[2]/div[2]/div[1]/table"
  2. XPath="/html/body/main/div/div[4]/div/div/div/div[3]/div[1]/div/div[2]/div[2]/div[2]/table"

Thank you so much!


let    // Define a list of slugs    Slugs = {"IBIT","FBTC","BITB","ARKB","BTCO","EZBC","BRRR","HODL","BTCW","GBTC"    },    // Define a function to construct URLs for each slug    ConstructURL = (Slug) => "https://ycharts.com/companies/" & Slug & "/total_assets_under_management",    // Define a function to apply transformation steps to each URL    TransformData = (URL) =>    let        Source = Web.Page(Web.Contents(URL)),        Data0 = Source{0}[Data],        Data1 = Source{1}[Data],        CombinedData = Table.Combine({Data0, Data1}),        // Extract the company name from the URL        CompanyName = Text.BetweenDelimiters(URL, "companies/", "/"),        // Rename columns dynamically based on the company name        RenamedColumns = Table.RenameColumns(CombinedData, {{"Date", "Date"}, {"Value", "Value_" & CompanyName}}),        // Change the data type of the columns        ChangedType = Table.TransformColumnTypes(RenamedColumns, {{"Date", type date}, {"Value_" & CompanyName, type text}})    in        ChangedType,    // Construct URLs for each slug    URLs = List.Transform(Slugs, each ConstructURL(_)),    // Apply transformation to each URL and combine the results    CombinedTables = List.Transform(URLs, each TransformData(_)),    // Combine new data with existing data    CombinedTable = if List.Count(CombinedTables) > 0 then Table.Combine(CombinedTables) else nullin    CombinedTable

Update: I've managed to put everything in two columns but I'd rather have one date column and ten columns with values:

let    // Define a list of slugs    Slugs = {"IBIT","FBTC","BITB","ARKB","BTCO","EZBC","BRRR","HODL","BTCW","GBTC"    },    // Define a function to construct URLs for each slug    ConstructURL = (Slug) => "https://ycharts.com/companies/" & Slug & "/total_assets_under_management",    // Define a function to apply transformation steps to each URL    TransformData = (URL) =>    let        Source = Web.Page(Web.Contents(URL)),        Data0 = Source{0}[Data],        Data1 = Source{1}[Data],        CombinedData = Table.Combine({Data0, Data1}),        // Extract the company name from the URL        CompanyName = Text.BetweenDelimiters(URL, "companies/", "/"),        // Change the data type of the columns        ChangedType = Table.TransformColumnTypes(CombinedData, {{"Date", type date}, {"Value", type text}}),        // Add a custom column for company name        AddedCompanyColumn = Table.AddColumn(ChangedType, "Company", each CompanyName)    in        AddedCompanyColumn,    // Construct URLs for each slug    URLs = List.Transform(Slugs, ConstructURL),    // Apply transformation to each URL and combine the results    CombinedTables = List.Transform(URLs, each TransformData(_)),    // Combine new data with existing data    CombinedTable = if List.Count(CombinedTables) > 0 then Table.Combine(CombinedTables) else nullin    CombinedTable

Could someone please help me fix this VBA code so that the output doesn't have any null rows and all data for a particular date is displayed parallely? Also, it'd be great if you could optimize this macro (it takes sooo long for it to load) so that it doesn't download all data from those website but just accesses these two containers from each website to download tables:

  1. XPath="/html/body/main/div/div[4]/div/div/div/div[3]/div[1]/div/div[2]/div[2]/div[1]/table"
  2. XPath="/html/body/main/div/div[4]/div/div/div/div[3]/div[1]/div/div[2]/div[2]/div[2]/table"

Thank you so much!


Viewing all articles
Browse latest Browse all 18095

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>