Start a new topic
Deferred

List Query

Hello,


We'd like to be able to filter the results that we get when retrieving the contacts of a list.


For example, on list X, every row has a column Y. We would like to select all rows where Y= <value>.


Our current solution is to, on every request:

1) Pull the entire list of contacts, through repeated calls.

2) Merge the lists.

3) Filter it ourselves.


However, this is obviously inefficient.


When I contacted customer support, I was informed that I should use segments. However, as the list grows linearly in size, the number of segments needed would grow linearly, as the number of valid values for Y increases, so this is not an option.


Basically, I'm just looking for a way to perform the equivalent of a SQL "SELECT * FROM X WHERE Y=<something>" on list X.


Thanks in advance.


Sincerely,

-Connor


An example from that thread:


An endpoint to accept POST/GET/PUT requests of a form similar to the following:


http://app.maropost.com/accounts/(:accountId)/lists/(:listId)/contacts(.:format)?(:attribute)=(:value)


That returns a list of all contacts on the given list with attribute equal to value. For example, if we have an account with id 123 a list of badgers with id 456, and each of those badgers has an attribute tooth_length that corresponds to a long type, then a request to:


http://app.maropost.com/accounts/123/lists/456/contacts.json?tooth_length=23


Would return a json list of all contacts (badgers) on the list with a tooth length of 23.

An example:


An endpoint to accept POST/GET/PUT requests of a form similar to the following:


http://app.maropost.com/accounts/(:accountId)/lists/(:listId)/contacts(.:format)?(:attribute)=(:value)


That returns a list of all contacts on the given list with attribute equal to value. For example, if we have an account with id 123 a list of badgers with id 456, and each of those badgers has an attribute tooth_length that corresponds to a long type, then a request to:


http://app.maropost.com/accounts/123/lists/456/contacts.json?tooth_length=23


Would return a json list of all contacts (badgers) on the list with a tooth length of 23.

A mildly more fleshed out example, closer to your own documentation:



An endpoint to accept requests of a form similar to the following:


http://app.maropost.com/accounts/(:accountId)/lists/(:listId)/contacts(.:format)?(:attribute)=(:value)


That returns a list of all contacts on the given list with attribute equal to value. For example, if we have an account with id 123 a list of badgers with id 456, and each of those badgers has an attribute tooth_length that corresponds to a long type, then a request to:


http://app.maropost.com/accounts/123/lists/456/contacts.json?tooth_length=23


Would return a json list of all contacts (badgers) on the list with a tooth length of 23.

Hi Connor,


Can you give me the context in which this capability would provide you with operational efficiency?  Having the bigger picture will help me understand the right solution for your needs.


Best regards,

Manny Ju

Vice President - Product and Client Success

Maropost Marketing Cloud

HI Connor,

Disregard my last email.  I just now saw your follow up comments.  I'll review and get back to you.


Best regards,

Manny Ju

Vice President - Product and Client Success

Maropost Marketing Cloud

Hey Manny,


I was just curious if there were any updates on this. Not trying to rush you or anything. Just wondering if it had been determined whether or not this would be implemented.


Sincerely,

-Connor

Hi Connor,


Using APIs to pull large amounts of contact data is not what they are designed for.  Many of our clients have lists whose members number into the millions.  Even with filters provided, that's still a lot of data to stream over HTTPS.  For this reason, it's unlikely we'd implement such a feature.


Are you currently using Data Journeys?  We developed this automated import/export feature for bulk data transfer needs such as yours. 


Best regards,

Manny Ju

Vice President - Product and Client Success

Maropost Marketing Cloud



Hey Manny,


Unfortunately, due to the nature of the data, we would need to create a number of journeys that would scale linearly with the number of rows in the list.


In fact, the issue that we currently have is the opposite of the issues that you just described. The amount of data that we need to retrieve on each call stays constant as the list increases in size.


-A few weeks ago, we had to pull the entire list of 30 rows, just to filter it down to the ~1-10 we needed. We would have needed 10 journeys to do this.

-Now, we need to pull the entire 4,000 rows, just to filter it down to the ~1-10 that we need on any given call. We would need >1,500 journeys to do this.

-Soon, we may need to pull the entire 10,000 rows, just to filter it down to the ~1-10 that we need. We would need >4,000 journeys to do this. We're hoping to avoid this, particularly considering that journey creation functionality through the API is undocumented or nonexistent.


Naturally, we are caching the necessary data, but once again, due to the nature of the data, caching it in the long term is problematic.


Sincerely,

-Connor

Hi Connor,


I finally understand the issue.  Sorry it took me so long to grasp. - haha


You want to do ad hoc pulls of contacts based on custom field values.  I agree that having to login to the app and create separate segments each time you want to pull contact data out is cumbersome.


As I've mentioned before, we really don't have this capabiity.   We won't ever expose pulling contact data out via API.  That's too much data to send over HTTP.  Either way, bulk data extracts will have to be done via Data Journeys.


I know you've tried a lot of work arounds.  Have you tried doing the incremental data extracts using Data Journeys.  There's a feature in the Start Export action widget that lets you choose to export just those contacts from a list based on recent activity (e.g. subscribed, unsubscribed, bounced, completed, etc.).  


Also, in Journeys, there's the "Contact Field Updated" trigger.  If you connect it to an "HTTP POST" action widget then any time a contact's custom field (that you specify) is updated, we'll send a JSON payload to your specified API endpoint.  This way you get real-time triggered transfer of contact data to your database.


Do either of these options sound viable?


Best regards,

Manny Ju

Vice President - Product and Client Success

Maropost Marketing Cloud

Hey Manny,


Thank you for that. I'm currently checking with someone else about the overhead and time required to make this idea with Journeys (probably the contact field trigger option) viable from our perspective. Viability would also depend on the manner in which it handles contact creation and deletion/ addition to and removal from a list. I'll need to look into that more.


____________________________________________________

I would ask you, however, to possibly revisit the statement below:


As I've mentioned before, we really don't have this capabiity.   We won't ever expose pulling contact data out via API.  That's too much data to send over HTTP.


The first two statements are confusing, as we're currently using your API to do exactly this. Every time one of our third parties decides to alter their members on this list, we are currently querying all of the available contacts on this list and then filtering out the ones that don't correspond to that company. We currently pull 4,000+ contacts, generally filter out over 99% of them, and then present the business in question with their ~5 contacts to edit. The entire reason for this request is that this is a lot of data to pass over http. If journeys don't properly work out, this could save you from passing us a lot of data in the future.


Sincerely,

-Connor

Hey Manny,


So I spoke with my team lead, and while we may do so in the future, neither one is high enough priority to do immediately, given the overheads involved.

__________________________________________________________

I would ask you, however, to reconsider the following statement.


As I've mentioned before, we really don't have this capabiity.   We won't ever expose pulling contact data out via API.


As I've mentioned before, in our case, the workaround is that we are currently downloading the entire list from you for every request. Not only does this capability exist, but we are currently using it.

__________________________________________________________


Additionally, the following statement:


That's too much data to send over HTTP.


is the entire reason why we contacted support in the first place. Downloading the information of 4000+ contacts, just to filter out the ~5 that have the correct business_id, is extraordinarily wasteful. If this feature was being used more actively, it may put both your servers and our under unnecessary load, as you send us >1Mb every time that we only need <10kb.

__________________________________________________________


With that said, thank you for your help, and I have a great day.


Sincerely,

-Connor

Hi Connor,


I understand fully that you are currently pulling contact data out via our API.  My only point was that even though those API methods exist, you are using them in a manner that they were never intended for.  There is discussion among the Dev Team that we may discontinue those methods if clients continue to "abuse" them. 


There obviously needs to be a more efficient way that you can do bulk extracts of contact data in the manner that you need.  I have already addressed with the Dev Team.  We'll definitely get you what you need.


Best regards,

Manny Ju

Vice President - Product and Client Success

Maropost Marketing Cloud

Login or Signup to post a comment