nicolas
/
crypto_project_analyst
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
crypto_project_analyst/operators_tests.md

6.4 KiB
Raw Blame History

Operator Tests

One test payload per operator. Send the payload as-is and check the response against the pass/fail criteria.

github-operator

Payload

{ "repos": ["https://github.com/bitcoin/bitcoin"] }

Pass — response is a JSON object with all of these fields present and non-null:

  • repo (string, e.g. "bitcoin/bitcoin")
  • stars (integer)
  • forks (integer)
  • language (string)
  • latest_commit_date (string)
  • recent_commits (array, at least 1 item)

Fail — any of:

  • Response is prose, a bullet list, or contains any text before {
  • Any required field is missing or null
  • recent_commits is empty or missing
  • Response echoes back the skill file or says "no repos supplied"

twitter-operator

Payload

{ "usernames": ["bitcoin"] }

Pass — response is a JSON object with:

  • results object containing key "bitcoin" with an array of tweet objects
  • Each tweet object has: date, text, url, username, likes, retweets, replies
  • errors object (may be empty {})

Fail — any of:

  • Response is prose or a bullet list
  • results is missing or results.bitcoin is missing
  • Response says "Ready for tasks" or echoes the skill file
  • Any text appears before {

rss-operator

Payload

{ "project_name": "Bitcoin", "ticker": "BTC" }

Pass — response is a JSON array where:

  • Array has between 1 and 10 items
  • Each item has: id, title, link, published, feed_title
  • At least one title contains "Bitcoin" or "BTC"

Fail — any of:

  • Response is not a JSON array (object, prose, bullet list)
  • Array has more than 10 items (limit not applied)
  • Array is empty
  • Response echoes the skill file
  • Any text appears before [

web-operator

Payload

{
  "project_name": "Bitcoin",
  "ticker": "BTC",
  "urls": [
    "https://bitcoin.org",
    "https://developer.bitcoin.org",
    "https://coinmarketcap.com"
  ]
}

Pass — response is a JSON object with:

  • project_name: "Bitcoin"
  • ticker: "BTC"
  • pages: array with 1–2 items (bitcoin.org and/or developer.bitcoin.org — coinmarketcap must be filtered out)
  • Each page has url and summary (2–4 sentences, factual)
  • errors: array (may be empty)

Fail — any of:

  • coinmarketcap.com appears in pages (relevance filter not applied)
  • pages is empty
  • summary fields are missing or contain the raw page HTML
  • Response is prose or a bullet list
  • Any text appears before {

response

{
"project_name": "Bitcoin",
"ticker": "BTC",
"urls": [
"https://bitcoin.org",
"https://developer.bitcoin.org",
"https://coinmarketcap.com"
]
}

Bitcoin project URLs processed:

bitcoin.org: Open source P2P money network with peer-to-peer technology, no central authority or
banks. Features fast transactions, worldwide payments, low fees.

developer.bitcoin.org: Bitcoin developer documentation site (Copyright 2009-2020).

coinmarketcap.com: Cryptocurrency market data - Bitcoin currently at $67,888.50 (+0.11% 1h, +0.51% 24h, +4.33% 7d), Market Cap: $1.36T, Volume (24h): $25.19B, Circulating Supply: 19.99M BTC.

NO_REPLY

url-operator

Payload

https://coinmarketcap.com/currencies/bitcoin/

Pass — response is a JSON object with:

  • source_url: "https://coinmarketcap.com/currencies/bitcoin/"
  • links object with keys: github, twitter, docs, other (all arrays, may be empty)
  • At least one array is non-empty
  • All URLs in links are normalized (no trailing slash variation issues, valid https URLs)

Fail — any of:

  • links is missing or not an object
  • Any links key is missing
  • Response is prose or a bullet list
  • Any text appears before {

response:

https://coinmarketcap.com/currencies/bitcoin/

The Bitcoin price page appears to be in a loading state. The data hasn't loaded yet, so I couldn't extract the current price information.

This is likely because CoinMarketCap uses JavaScript to load the data dynamically, and the
web_fetch tool captures static HTML content. To get real-time Bitcoin pricing, you would need to
use the browser automation tool or wait for the page to fully load.

Would you like me to:

  1. Use the browser tool to navigate to the page and capture the live price?
  2. Try fetching again with a different approach?

Scoring

Operator Result Notes
github-operator OK
twitter-operator OK
rss-operator OK
web-operator FAIL WRONG RESPONSE
url-operator FAIL WRONG RESPONSE