#74 new
shopoto

videos are retrieve with force fetch but are not with cron...?

Reported by shopoto | July 29th, 2008 @ 10:03 AM

Hi grand master Guillermo...! ;-)

There is a strange behavior on my install...

I've a wordpress 2.6 running with wp-o-matic 1.0RC4-6.

I've hacked the function fetchFeed in order to retrieve videos from youtube, dailymotion etc...

Here is the hack i followed :

function fetchFeed($url, $stupidly_fast = false, $max = 0)

{

  1. SimplePie

if(! class_exists('SimplePie'))

require_once( WPOINC . 'simplepie/simplepie.class.php' );

$feed = new SimplePie();

/********************************************/

// Remove these tags from the list

$strip_htmltags = $feed->strip_htmltags;

unset($strip_htmltags[array_search('object', $strip_htmltags)]);

unset($strip_htmltags[array_search('param', $strip_htmltags)]);

unset($strip_htmltags[array_search('embed', $strip_htmltags)]);

$strip_htmltags = array_values($strip_htmltags);

$feed->strip_htmltags($strip_htmltags);

/********************************************/

$feed->enable_order_by_date(false); // thanks Julian Popov

...

...

...

}

This works quite well when i do a forcefetch from the wp-o-matic dashboard.

But when my external feeds are fetch from the cron job.

I do not know why but

I do retrieve the different posts comming from the feeds i have configured

but all the videos there were on theses post are removed...!

I really do not understand why???

is there a way to create a special cron job with the "forcefetch" mode activated?

If needed I'll configure as many force fetch cron jobs as many campaign i have.

Thanks by advance for your answer!

Comments and changes to this ticket

  • Guillermo Rauch

    Guillermo Rauch July 29th, 2008 @ 01:50 PM

    Cron behaves almost exactly like forcefetch does... That's strange.Plus,

    there's no function that 'removes' posts (except the one in the tools tab).

    Have you tried running the cron URL directly from your

    browser?

  • shopoto

    shopoto July 29th, 2008 @ 07:23 PM

    As you ask I try to paste the regular cron url on ff3.

    (I manage to have only one campaign containing only one feed contening the last posts, 1 or 2 post containing vidéos, to be retrieved)

    I Launch the url then I check my front page...

    And I correctly retrive the post with all the video inside...

    So everything seems fine.

    But as I were sure there a problem.

    I reset this campaign,

    remove the post created from this test campaign

    and even delete records from the table "wpo_campaign_post" corresponding to the se previous retrieved post.

    And I try to lauch the cron from webcron.org.

    Once I correctly configure the cron, I test it from their interface.

    (there is a "test cron" link to direct test without wainting...)

    The cron did retreive all the post i have previously retrieve (and then erased)...

    BUT as I said none of the video contained in these posts appeared.

    In fact when i look closely to the html code on the posts fetched, all the tags are removed...!

    To be sure of that I erase again these same post from this test campaign, delete again corresponding records from the table "wpo_campaign_post" and reset once agin the test campagn.

    This time I let the regular cron on the web server that host my wordpress blog...

    Results: The same strange behaviour...

    I retrieve all the last post which should contain vidéos but all the tags are removed in despite of the code I set on the wpomatic.php, and so the videos are also removed...

    So now I'm sure I Have not dream! lol ;-)

    Could it be possible that this strange behaviour could come from a kind of time out when wp-o-matic must deal with the "forced strip_htmltags"?

    He skip these forced tags, bypassing my order, to continue retrieving the rest of the post and other following posts.

    May be that webcron.org and my server have a too low "execution_time_limit" or something like that?

    Yet from my ff3 direct retrieving the last 2 post of one campaign don't take more then 2 or 3 seconds...

    Anyway webcron should have and my server have an execution time limit mich higher then 3 seconds...!

    So... I'm completely lost.

    Thanks by Advance for your answer...!

  • KBruno

    KBruno October 30th, 2008 @ 11:09 PM

    Hey Giillermo, First, I've got over 400 campaigns working beautifully on a single site with 2 minor fixes for using feed date & stopping runaway duplicate posts. That is except for certain html tags being stripped.

    As others, using the forcefetch displays posts with the embed, param or object tags intact. Not so otherwise. You said "Cron behaves almost exactly like forcefetch does..." Can you expand on what "almost" means?

    Can you also explain how the thing works when the cron isn't being used at all? What I mean is: I don't have a unix cron set for wpomatic & I don't run the cron URL directly but the campaigns are processed in perfect time with the Frequency set in the options. What's that about? The logs don't show that cron.php has run at all & I rarely forcefetch 1 or 2 campaigns & yet over 400 campaigns continue to be processed & updated.

    The reason I wonder about this is that I've gone so far as to remove the embed, param or object tags from both simplepie.class.php in wpomatic and simplepie.inc in simplepie-core in an attempt to get this resolved. So where does wpomatic process the feeds without cron.php being run & how does it strip tags that aren't in the strip_htmltags array?

    Any help would be huge.

  • KBruno

    KBruno November 20th, 2008 @ 04:21 PM

    I found a remedy for imports of most embed, param & object videos. First in wpomatic.php I added one key to the wp_insert_post array:

    $postid = wp_insert_post(array(
        'post_title'                => $title,
        'post_content'              => $content,
        'post_content_filtered'     => $content,
        'post_category'           => $category,
        'post_status'             => publish,
        'post_author'             => $authorid,
        'post_date'               => $date,
        'comment_status'          => $comment_status,
        'ping_status'             => $allowpings,
        // new key
        'post_wpom'             => yes
    ));
    
    

    Next in the function sanitize_post in /wp-includes/post.php I added one condition:

    function sanitize_post($post, $context = 'display') {

    if ( 'raw' == $context )
        return $post;
    if ( is_object($post) ) {
        foreach ( array_keys(get_object_vars($post)) as $field )
            $post->$field = sanitize_post_field($field, $post->$field, $post->ID, $context);
    } else {
        foreach ( array_keys($post) as $field )
            // changes referer only if called by wpomatic
            if(array_key_exists('post_wpom', $post)) {
               echo ' ';
            }
            $post[$field] = sanitize_post_field($field, $post[$field], $post['ID'], $context);
    }
    return $post;
    
    

    }

    That's it & it works on all the feeds I've tested (a few I've had to remove line breaks in the 'content' before the wp_insert_post so wordpress doesn't add '
    's but that's an error in the feed).

    I use the psuedo cron so I'd be interested to hear if it works in with other options.

  • tbone

    tbone December 12th, 2008 @ 10:29 AM

    I've just tried the above patch from KBruno but it still doesn't create posts with an embedded YouTube video, even though they are in the feed.

    Tried it with both a forced fetch and a pseudo cron.

    In fact it doesn't put anything at all where the video should be, it's just blank.

    Is WPOM supposed to support embedding videos or not? I have been looking but can't find a definitive answer. I guess from this ticket that it doesn't and various people are trying different methods to get it working.

    Would be nice to have an 'official' patch though.

  • tbone

    tbone December 12th, 2008 @ 10:40 AM

    Correction.

    I forgot to add the code from the first post.

    Now seems to work ok from both forced fetch and pseudo cron.

    Thanks guys!

  • Jean Mich

    Jean Mich January 13th, 2009 @ 04:00 PM

    I'm looking SimplePie doc (http://simplepie.org/wiki/) for try to change defaut width and height, does anyone know how to do ?

    Thx

  • Luke Page

    Luke Page February 3rd, 2009 @ 05:47 AM

    I've seen lots of problems to do with html stripping through all the comments on wp-o-matic and no one post that explains it all.. so I'll try and explain what I've discovered. (btw Guillermo - great product. You might want to put this text somewhere easier to find.

    There are 2 filters that will effect posts being imported.

    The first is simple-pie and if you look at the simple pie include php file and find strip_html then you could hack this to change what tags and attributes are removed.

    (Because it suits me, I've hacked, but there must be a cleverer way to get this sorted? Option on the next release?)

    The second filter is the WordPress filtered html filter. By default admin users don't filter, but a non-user does (See bottom of kses.php in the word press includes).

    If your logged in as admin and do a force fetch, the html filter hasn't been attached and nothing is filtered. When doing a cron however, no-one is logged in and when kses is initiated, the filter is attached.

    I got round it by disabling all html filtering as in my case I'm not bothered about having it, every user is trusted.

    If the filtered html could depend on the user you've set to insert the posts as (by checking that users unfiltered_html and disabling/enabling kses before calling wp_insert_post) and making the simple pie filtering a config then that would be a fantastic improvement to wp-o-matic.

  • l053r

    l053r March 7th, 2009 @ 07:55 PM

    I want to embed videos from youtube, dailymotion etc can anyone please pack all those code in a file or tell me where should i put those codes? I'm not familiar with PHP.

    thanks

  • Ricardo Rauch

    Ricardo Rauch April 15th, 2010 @ 08:50 AM

    • Assigned user changed from “Guillermo Rauch” to “Ricardo Rauch”
  • Broukw

    Broukw June 14th, 2023 @ 01:44 PM

    There are many different types of raised toilet seats available on the market today. Some are designed to be permanently attached to the toilet, while others are portable and can be easily removed when not in use. Some raised toilet seats also come with additional features like armrests or padded seats for added comfort and support. https://mrtoiletseat.com/best-padded-toilet-seat/

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

WP-o-Matic project.

Referenced by

Pages