menu

Insider: MTGO Automation and Analysis Week 4

Are you a Quiet Speculation member?

If not, now is a perfect time to join up! Our powerful tools, breaking-news analysis, and exclusive Discord channel will make sure you stay up to date and ahead of the curve.

Welcome back! Last week I discussed the data model I’m using to store data, so this week I can finally get to how I actually scrape the data and place it in the data model. I’ll also visit standard numbers once more so we can look at the delta from last week!

Automation:

We're going to dabble in a little bit of code now. I'm using a language called Apex which is really a derivative of Java localized to Salesforce, the product I use as my data warehouse software. I'll include the snippets of code, and then an explanation afterwards. If you aren't a coder, hopefully the descriptions will give you a good idea about how these sorts of technologies work. If you are a coder you can ignore the easy stuff.

As discussed previously, I want to store event data, and I can get that event data from Wizards. The initial target is the following URL, which will give us back a list of events over the last 14 days: http://www.wizards.com/handlers/XMLListService.ashx?dir=mtgo&type=XMLFileInfo&start=14

When your browser requests a URL, you get back some text which can be rendered as a web page if it's HTML, or viewed as a text file, XML file, pdf, etc depending on some definitions in the response to your request. URL requests are not restricted to browsers however. If you make a URL GET request in code, you get back the same text/xml/html you'd get in your browser, but you have the option on how to interpret the response programmatically. Let me show you with the first bit of code:

string url = 'http://www.wizards.com/handlers/XMLListService.ashx?dir=mtgo&type=XMLFileInfo&start=14';
HttpRequest req = new HttpRequest();
req.setEndpoint(url);
req.setMethod('GET');
Http http = new Http();
HTTPResponse res = http.send(req);

JSONParser parser = JSON.createParser(res.getBody());

Stepping through it line by line, you can see that I define the URL, define a new HttpRequest that I call req, I set the URL target for req as the URL, I set the type of request as a "GET" request, and then I create an HTTPResponse called res as the result of sending my HttpRequest req. The end result, is that the body of the HttpResonse res will look something like this:

[{"__type":"XMLFileInfo:#Wotc.Web.DataDefinitions","Date":"10\/23","Hyperlink":"6118427","Name":"Legacy Daily"},{"__type":"XMLFileInfo:#Wotc.Web.DataDefinitions","Date":"10\/23","Hyperlink":"6118429","Name":"Modern Daily"},...,...]

The last line takes the body from res and uses a JSON method to parse it. Boiling that down a bit, it makes the format of the returned information a little easier to work with in code by creating a list of smaller elements. Now let's break down the data even further:

MTGO_Event__c e;
Map Event_Insert_Map = new Map();

In this code block, I define a few things. The first is a new record e belonging to the event table. The second is a Map called Event_Insert_Map, which is like a list of records, except that every record has a unique handle that allows me to pull it back out of the list easily. Next, we are going to use the data we got from the first block of code:

while (parser.nextToken() != null) {
if(string.valueOf(parser.getCurrentToken()) == 'START_OBJECT'){
e = new MTGO_Event__c();
} else if (string.valueOf(parser.getCurrentToken()) == 'VALUE_STRING' && parser.getCurrentName() == 'Date'){
e.Event_Date__c = date.parse(parser.getText()+'/'+string.valueOf(date.today().year()));
} else if (string.valueOf(parser.getCurrentToken()) == 'VALUE_STRING' && parser.getCurrentName() == 'Name'){
e.Event_Type__c = parser.getText();
} else if (string.valueOf(parser.getCurrentToken()) == 'VALUE_STRING' && parser.getCurrentName() == 'Hyperlink'){
e.Event_Number__c = parser.getText();
e.name = e.Event_Number__c;
} else if (string.valueOf(parser.getCurrentToken()) == 'END_OBJECT' && (e.Event_Type__c.contains('Daily') || e.Event_Type__c.contains('Premier')) ){
Event_Insert_Map.put(e.Event_Number__c,e);
}
}

The first line creates a loop that says, while there is still another item in the JSON parser list keep doing all the things below me. The next few lines are all very repetitive. They look at the piece of text from the JSON parser list, and they do different things depending on the label of that item. If the label says START_OBJECT, create a new record. If the label says Date, Name, or Hyperlink, set the corresponding field in the record e to that value. After these lines have been processed, we should have something that looks like this:

MTGO_Event__c e
e.Event_Date__c = 10/23
e.Event_Type__c = Legacy Daily
e.Event_Number__c = 6118427

And finally, when the END_OBJECT label appears, we place e into the Event_Insert_Map map. We then repeat the cycle until all the items in the list have been exhausted. The final step is an easy one, and looks like this:

insert Event_Insert_Map.values();

We take the values of the Event_Insert_Map (all the records, without their corresponding handles) and we insert them into the database. Now we have persisted all the events from the last 14 days! Since I do this every day, I also have a check where I make sure not to make any duplicates. I do this overlap in case there is an error, so I have some time to correct the problem!

Next week I'll increase the complexity a little bit by using the list of events to scrape all the individual decks and store them as well.

Speculation:

Now, enough of that technical nonsense! I'm going to report on the standard environment again this week, so we can look at the delta between last weeks article and now, 7 days later. I spent a ton of time working on the styling of the tables and I'm also going from 50 rows down to 25 rows on each of the tables. The end result should be a much more refined and easy to read look. If you prefer it this way let me know.

Mythics:

Card Name Bot Buy Price Standard Quantity Last 7 Days Standard Quantity Previous 7 Days Quantity Delta Standard Decks Last 7 Days Standard Decks Previous 7 Days Deck Delta
Jace, Architect of Thought 25 487 463 24 193 202 -9
Master of Waves 6.7 343 396 -53 87 99 -12
Thassa, God of the Sea 6.85 328 375 -47 85 98 -13
Erebos, God of the Dead 6.4 256 195 61 246 174 72
Sphinx's Revelation 36.5 223 151 72 68 49 19
Elspeth, Sun's Champion 11.5 180 179 1 110 126 -16
Polukranos, World Eater 7.55 170 191 -21 48 55 -7
Blood Baron of Vizkopa 12.9 149 193 -44 78 109 -31
Stormbreath Dragon 13.9 141 128 13 43 42 1
Domri Rade 20.25 136 189 -53 42 60 -18
Garruk, Caller of Beasts 14.5 135 171 -36 37 50 -13
Xenagos, the Reveler 8.35 124 114 10 63 67 -4
Chandra, Pyromaster 17.1 107 105 2 56 63 -7
Jace, Memory Adept 4.05 83 54 29 53 34 19
Obzedat, Ghost Council 10.2 69 155 -86 31 65 -34
Nylea, God of the Hunt 2.95 68 106 -38 38 46 -8
Purphoros, God of the Forge 4.65 61 51 10 30 23 7
Ashiok, Nightmare Weaver 7.1 54 53 1 24 20 4
Voice of Resurgence 25.5 34 58 -24 9 15 -6
Vraska the Unseen 5.2 28 13 15 18 7 11
Liliana of the Dark Realms 3.3 17 30 -13 12 17 -5
Ajani, Caller of the Pride 4.3 14 44 -30 9 24 -15
Rakdos's Return 7.7 14 59 -45 8 36 -28
Heliod, God of the Sun 2 10 13 -3 6 7 -1
Kalonian Hydra 6.5 6 13 -7 4 6 -2

Rares:

Card Name Bot Buy Price Standard Quantity Last 7 Days Standard Quantity Previous 7 Days Quantity Delta Standard Decks Last 7 Days Standard Decks Previous 7 Days Deck Delta
Mutavault 14.6 965 972 -7 340 353 -13
Nightveil Specter 2 934 662 272 238 176 62
Thoughtseize 6.65 884 798 86 262 244 18
Hero's Downfall 6.75 865 752 113 245 226 19
Desecration Demon 3.75 723 683 40 187 178 9
Underworld Connections 1.35 636 497 139 208 203 5
Pack Rat 0.1 538 254 284 267 121 146
Lifebane Zombie 5.3 526 576 -50 194 183 11
Nykthos, Shrine to Nyx 3.6 475 456 19 266 227 39
Mizzium Mortars 3.4 442 512 -70 152 190 -38
Boros Reckoner 7.5 431 506 -75 114 132 -18
Temple of Deceit 1.85 414 275 139 147 82 65
Ash Zealot 2.1 384 375 9 97 95 2
Temple of Silence 1.5 380 463 -83 120 136 -16
Chandra's Phoenix 1.6 359 385 -26 93 100 -7
Whip of Erebos 0.86 339 299 40 196 186 10
Tidebinder Mage 0.66 331 391 -60 85 101 -16
Pithing Needle 2.05 315 261 54 228 192 36
Hallowed Fountain 3.3 287 202 85 73 51 22
Supreme Verdict 1.65 278 195 83 82 51 31
Godless Shrine 3.3 268 357 -89 68 90 -22
Detention Sphere 1 265 176 89 79 63 16
Ratchet Bomb 0.4 258 342 -84 135 183 -48
Firedrinker Satyr 0.24 237 193 44 65 53 12
Burning Earth 1.7 229 261 -32 85 96 -11

My thoughts:

  • Voice of Resurgence fell in number of copies played, and busted through it's 30 ticket floor. I'm definitely looking to buy some sets of this card, but I'm hoping for a floor under 25. Going to watch the prices over the next few days and make a decision if this card goes no lower.
  • Nightveil Specter is putting up really strong numbers with a very low price for a rare of that quantity. Obviously the best time to buy has already past, but if you are late to the party there might still be room to go in. If you have copies, the strong performances should support a continued rise, so hold!
  • Master of Waves and Thassa, God of the Sea are still dipping in price but have maintained their position in the field. The floor should be any day now, with redemption opening very soon!

Conclusion:

Feel free to point out any of your own conclusions in the comments, I love hear great ideas. Thanks for the feedback last week on formatting, I hope things are much easier to read now!

One thought on “Insider: MTGO Automation and Analysis Week 4

  1. Awesome article and I love the data. I appreciate you color coding the table. It really highlights the changes in the cards and though I lack much coding background (I wish I knew what any of that code meant) the data you provide is incredibly useful and a great way to show trends that will allow anyone (paper or MTGO) to make predictions on future prices. I know I’ll be reading these articles every week.

Join the conversation

Want Prices?

Browse thousands of prices with the first and most comprehensive MTG Finance tool around.


Trader Tools lists both buylist and retail prices for every MTG card, going back a decade.